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Rppsrtt  frt  tlio  most  suttiorititlvo  ond  most  cirotuttir  constdorod  products  tOA  publlihos. 
Tbop  noffliotlp  ombody  rosolts  ol  ms|or  projocts  which  (a)  ha«o  a  direct  boaring  on 
dacisluns  sHoctlng  nia|or  programs,  (b|  address  Issuas  ol  slgnlllcant  conesrn  to  tha 
Eiacullva  Branch,  the  Congress  and/or  tha  public,  or  (c)  address  Issuas  that  ha«a 
slgnlllcant  economic  Implications.  IDA  Reports  are  rayiowod  by  outside  panels  ol  aiperts 
to  ansura  Ihoir  high  quality  and  ralavanca  to  tha  problomi  studied,  and  they  am  ralaasad 
by  tha  President  ol  IDA. 

Group  Reports 

Group  Reports  record  the  lindings  and  results  ol  IDA.  established  working  groups  and 
panels  composed  ol  senior  Individuals  iddressing  major  Issues  which  otherwise  would  be 
the  subjs't  ol  an  IDA  Report.  IDA  Group  Reports  ate  reviewed  by  the  senior  Individuals 
responsible  lor  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  and 
relevance  to  the  problems  studied,  and  are  released  by  tha  President  ol  IDA. 

Paperi 

Papers,  also  authoritative  and  earelully  considered  products  ol  IDA.  address  studies  that 
are  narrower  In  scope  than  those  covered  In  Reports.  IDA  Papers  are  reviewed  to  ensure 
that  they  meet  the  high  standards  oipectad  ol  relareed  papers  In  prolesslonal  journals  or 
lormal  Agency  reports. 

Documentt 

IDA  Documents  are  used  lor  the  convenience  ol  the  sponsors  or  the  analysts  (a)  to  record 
substantive  work  done  In  quick  reaction  studies,  (b)  to  record  the  proceedings  ol 
conlerences  and  meetings,  (c)  to  make  available  preliminary  and  tentative  results  ol 
analyses,  (d)  to  record  data  developed  In  the  course  ol  an  Investigation,  or  (a)  to  lorward 
inlormallon  that  is  essentlsity  unanalynd  and  unevsluated.  The  review  ol  IDA  Documents 
Is  suited  to  their  content  sad  Intended  use. 


The  work  reported  in  this  document  wes  conducted  under  contract  MDA  903  <9  C  0003  lor 
the  Department  el  Delense.  The  publication  ol  this  IDA  document  does  not  Indicate 
endorsement  by  the  Department  ol  Delense,  nor  should  the  contents  be  construed  es 
reflecting  the  ettlctel  position  ot  that  Agency. 


UNCLASSIFIED 


REPORT  DOCUMENTATION  PAGE 

Form  Approvdd 

OUB  No.  0704-0183 

P>i,*c  to'w.  WI.CMn'o(  1.  MMIM  W  1  hour  p«  >MoanM.  OKhiOlng  -h.  tm.  lor  l..<*«>hg  »«rualom.  M.rtf.tra  «»  louroM,  g..n.ru,s  .,.o 

nwltoolnu^  Iho  ilol*  iMdob.  tnd  oonwloti^  or*)  rovlowmg  tw  cBiwllon  ol  InlonMtIon.  SomJ  oommont  togotang  tlV«  tordon  Mllmat  of  ony  otlar  upM  U  Ihio  ooloaten  o<  rrrlormotion. 

OiOudng  •I.ggooiuno  (or  roduiAig  IN*  bur*tv  «>  WaoNrvioh  Hoadauonon  SoivtoM.  OMOorolo  tot  toloiThOtlon  t^rorlono  ond  noporu.  Ill S  Jottoroori  Oovto  Hlgriooy,  SiOw  1104.  Atlitiglon. 

1.  AGENCY  USE  ONLY  J.  BEPOnTDAlE  1  REPORT  TYPE  AND  DATES  C»«EREO 

April  1992  Final  Report,  J’tn  1991  -  Apr  1992 

4.  PTLE  AND  SUePTLE 

Simulators  and  Enhanced  Training 

S.  FUNDING  NUMBERS 

C-MDA-903-89C-0003 

T-L7-798 

t.  AUTHORCS) 

Bruce  N.  Angier,  Earl  A.  Alluisi,  and  Stanley  A.  Horowitz 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADORESS<ES) 

Institute  for  Defense  Analyses 

1801  N.  Beauregard  Street 

Alexandria,  VA  22311-1772 

t.  PERFORMING  ORGANIZI4TION 

REPORT  NUMBER 

IDA-P-2672 

9.  SPONSORINWUONITORMO  AGENCY  NAME(S)  AND  ADORESSfES) 

OASD(FM&P) 

Room  3B930,  The  Pentagon 

Washington,  D.C.  20301 

ia  SPONSORINU/MONITORING 

AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12A.  OiSTRiaunON/AVAlLASIUTV  STATEMENT 

Approved  for  public  release;  distribution  unlimited. 

12B.  DBTRIBUnON  COC€ 

IS  ABSTRACT  ^Ateirimum 200  naxMl 

In  this  paper  we  discuss  simulators  and  training  devices  in  the  overall  context  of  training  and  show 
how  their  cost-effectiveness  might  be  quantified.  The  paper  discusses  current  DoD  policies  on  training 
and  simulation  and  briefly  describes  existing  practice.  The  paper  summarizes  selected  cost-effectiveness 
analyses  from  research  literature  comparing  simulators  and  training  devices  with  actual  equipment  a'nd 
operations,  reinterprets  some  of  the  dm  behind  these  analyses,  and  collects  additional  cost  information, 
liie  results  of  this  examination  suggest  that  all  major  existing  training  options  have  value  and  that  laige 
changes  in  the  simulator  mix  should  not  be  made  suddenly.  However,  on  the  margin,  additional  1 

simulators  and  training  devices  are  affordable,  and  in  many  situations  appear  to  be  cost-effective  relapve 
to  actual  equipment  and  operations.  The  analyses  are  not  conclusive  due  to  incomplete  data.  1 

Recommendations  are  ma^  on  how  to  develop  additional  information.  \ 

14.  SUB  JEST  TERMS 

Costs,  Simulators,  Military  Training 

IS  PRICE  CODE  1 

17.  SECURITY  CLASSIFICATION  It.  SECURITY  CLASSTCATION  It.  SECURITY  CLASSIFICATION 

OFREPORT  OPTHBPAOB  OfABSTRACT 

Unclassified  Unclassified  Unclassified 

2a  UMITATIONOP 

ABSTRACT 

SAR 

NSN  754001-280-5500 


Standard  Form  298  (Rev.  2-89) 
PracHM  by  ANSI  SM.  Z»ia 

2M-Ias 


UNCLASSIFIED 


IDA  PAPER  P-2672 


SIMULATORS  AND  ENHANCED  TRAINING 


Bruce  N.  Angler 


Afpmti  for  ooblie  roliiio:  OMrtlHitloii  onllniIttO. 


IDA 

INSTITUTE  FOR  DEFENSE  ANALYSES 

Contract  MDA  903  89  C  0003 
Task  T-LT-T^S 


PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the  Office 
of  the  Assistant  Secretary  of  Defense  (Force  Management  and  Personnel)  (OASD 
(FM&P)),  under  contract  MDA  903  89  C  0003,  Task  Order  T-L7-798,  issued  15  March 
1990.  The  objective  of  the  task  was  to  identify  promising  approaches  to  maintaining  strong 
military  manpower  capability  during  a  period  of  declining  budgets  and  force  levels.  This  is 
one  of  seven  papers  to  be  published.  Each  paper  covers  a  specific  area  of  nrilitary 
manpower  management:  the  proper  experience  mix,  personnel  movement,  the  timing  of 
training,  lateral  entry,  the  link  between  career  progression  and  assumption  of  management 
responsibilities,  individual  training  methods,  and  increased  use  of  simulators  for  training. 
This  paper  addresses  the  use  of  simulators  for  training. 

This  work  was  reviewed  by  Waynard  C.  Devers,  J.  D.  Fletcher,  and  William  T. 
Mayfield  of  IDA 


EXECUTIVE  SUMMARY 


This  summary  has  three  sections,  llie  first  summarizes  the  tasking  that  this  paper 
addresses.  The  second  section  outlines  the  organization  of  the  paper,  and  the  third  section 
summarizes  the  main  body  of  the  paper. 

INTRODUCTION 

The  issue  of  whether  the  increased  use  of  individual  and  networked  simulators  and 
training  devices  for  training  should  be  undertaken  was  chosen  for  study  because  of  a 
perception  that:  (a)  in  some  cases  increased  use  of  simulator-based  training  is  feasible  or 
more  cost-effective  than  current  training  practices,  and  (b)  there  is  increased  pressure  on 
training  budgets.  The  purpose  of  this  paper  is  to:  (a)  provide  a  framework  to  assess  the 
cost-effectiveness  of  individual  and  networked  simulators  and  training  devices  to  provide 
peacetime  training  for  warfighting  skills,  (b)  report  or  perform  preliminary  analyses  within 
this  framework,  and  (c)  draw  conclusions  from  these  analyses,  for  either  actions  to  take  or 
information  to  gather. 

Keeping  the  scope  of  this  paper  to  simulators  is  difficult.  Most  peacetime  military 
activity  can  be  looked  on  as  preparation  for  warfighting,  and  much  of  that  preparation  is 
simulation  in  a  general  sense.  Additionally,  technological  and  operational  trends  are 
blurring  distinctions  and  allowing  interoperability  between  simulators,  other  training 
devices,  war  games,  equipment  stimulators,  and  training  on  instrumented  and 
uninstrumented  ranges.  We  believe  we  have  maintained  these  distinctions  where 
appropriate,  but  will  not  hesitate  to  use  “simulators”  to  mean  several  of  these  categories, 
particularly  “simulators  and  'mining  devices.” 

OUTLINE 

This  paper  is  divided  into  an  introduction  and  four  substantive  sections.  The  first 
two  substantive  sections  provide  background.  The  first  of  these,  Basic  Issues,  describes  a 
taxonomy  for  viewing  training,  a  brief  description  of  simulation  in  training  and  an  approach 
to  measuring  the  cost-effcctiveness  of  various  methods  of  training.  The  second.  Current 
Policies  and  Practices,  discusses  existing  Department  of  Defense  (DoD)  Directives  as  they 
apply  to  simulation  in  training.  Additionally,  it  describes  some  of  the  simulations  and 


simulators  that  are  currently  used  to  train  warfighting  capability  and  the  DoD  organizations 
that  monitor  their  use. 

The  Data  and  Analyses  section  has  three  purposes.  The  first  is  to  present 
information  on  current  expenditures  for  training  expenditures  related  to  operating  tempo 
(OPFEMPO).  The  second  is  to  review  literature  on  simulator  cost  and  effectiveness.  The 
third  is  to  attempt  cost-effectiveness  comparisons:  (a)  of  OPTEMPO  flight  hours  versus 
simulator  time,  and  (b)  of  Simulator  Network  (SIMNET)-like  facilities  with  exercises  at 
both  the  National  Training  Center  (NTC)  and  home-station  training  ranges  and  maneuver 
areas.  The  latter  comparison  cannot  be  made  because  all  the  necessary  cost  and 
effectiveness  data  are  not  available.  However,  several  useful  inferences  can  be  drawn 
about  policy  options  and  further  research. 

The  Conclusions  and  Recommendations  section  summarizes  key  conclusions  from 
our  research  and  discusses  actions  to  take  and  information  to  gather  that  could  further 
improve  future  training  decisions. 

SUMMARY 

Basic  Issues 

There  is  sufficient  evidence,  both  anecdotal  and  quantifiable,  to  indicate  that  combat 
experience  is  the  best  trainer  of  combat  skills.  Simulations  have  long  been  used  as 
peacetime  substitutes  for  combat  operations.  In  many  cases,  this  has  been  done  because 
training  through  actual  exercises  could  not  be  done.  In  other  cases,  the  reason  was  that 
simulations  were  more  cost-effective.  Individual  and  networked  simulators  and  training 
devices  are  one  simulation  approach. 

We  now  have  both  (a)  the  technological  capability  to  create  a  continuum  of  training 
options  based  more  heavily  on  computer  simulations  and  (b)  conceptually  sound  methods 
to  measure  the  cost-effectiveness  of  these  approaches.  These  technological  capabilities 
include  better  individual  simulators  and  networked  simulators,  such  as  SIMNET,  for 
collective  training.  Replicating  existing  capabilities  for  individual  or  networked  computer 
simulators,  as  well  as  other  computer  simulations,  is  becoming  significantly  less  expensive 
due  to  reductions  of  25%  per  year  in  the  price  of  the  computer  equipment  that  is  often  used 
in  their  construction.  Additional  capabilities  are  being  developed  to  enhance  training  or  to 
support  other  requirements  such  as  mission  rehearsal. 

Both  qualitative  and  quantitative  cost-effectiveness  comparisons  can  be  made 
between  different  simulations  and  between  simulations  and  actual  operations.  Qualitative 


comparisons  usually  focus  on  the  realism  of  various  options  as  evaluated  by  the 
participants.  Quantitative  comparisons  use  methods  such  as  training  Transfer  Effectiveness 
Ratios  (TERs).  A  TER  is  the  ratio  of  two  Gaining  options’  contribution  to  meeting  some 
training  criterion.  This  TER  is  then  compared  with  the  ratio  of  the  costs  of  the  training 
options.  The  quantitative  comparison  follows  the  approach  of  microeconomic  theoiy  to 
arriving  at  the  most  efficient  combination  of  inputs  (time  spent  training  on  both  simulators 
and  actual  equipment)  to  produce  a  given  output  (individual  and  collective  readiness  or 
warflghting  usability). 

Current  Policies  and  Practices 

DoD  Directive  1322.18,  Military  Training,  and  DoD  Directive  1430.13,  Training 
Simulators  and  Devices,  contain  basic  guidance  for  using  simulators  in  training.  They 
direct  that  there  be  a  systems  approach  to  training  and  that  simulators  and  training  devices 
should  be  subject  to  a  cost-benefit  analysis  as  one  alternative  in  an  overall  training  strategy 
to  optimize  the  operational  readiness  of  the  force. 

The  primary  reason  for  using  simulators  is  improving  readiness,  but  cost  savings 
due  to  increased  simulator  use  are  an  important  secondary  consideration.  Guidance  for 
collective,  networked  simulators  is  not  explicit,  though  vigorous  programs  in  training 
technology  research  are  suggested. 

There  are  many  training  options  that  provide  simulations  of  actual  combat  These 
include  a  variety  of  exercises  (REFORGER,  NTC,  field  exercises  on  home-station  training 
areas),  simulators  (SIMNET,  individual  platform  simulators)  and  higher  echelon  war 
games.  These  options  provide  environments  that  train  through  integrating  skills  and 
knowledge  gained  previously,  offer  different  degrees  of  realism  and  effectiveness,  and 
have  different  costs. 

Data  and  Analyses 

We  develop  a  rough  estimate  of  the  cost  of  OPTEMPO,  one  of  the  primary  ways 
collective  training  is  provided.  In  FY  1991  dollars,  for  DoD,  this  estimate  is  $21  billion, 
including  $12  billion  of  flight  OPTEMPO,  $8  bilUon  of  ship  OPTEMPO,  and  $1  billion  for 
Army  vehicle  OPTEMPO.  Simulations  can  leverage  these  expenditures  to  increase  their 
efficiency  and  impact  on  readiness  and  effectiveness. 

The  Analyses  subsection  presents  two  types  of  comparisons  of  the  costs  and 
effectiveness  of  several  of  the  training  options.  The  first  shows  the  tradeoff  of  flying-hour 
OPTEMPO  for  possible  increases  in  simulator  time.  The  second  attempts  to  compare  the 
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costs  and  effectiveness  of  SIMNET  with  actual  exercises  at  field  training  locations.  The 
data  that  underlie  the  first  comparison  are  mainly  from  a  literature  review  of  maintenance 
and  flight  simulator  costs  and  effectiveness,  and  the  second,  from  studies  comparing 
SIMNET  and  field  exercises. 

The  purpose  of  these  examples  is  to  highlight  key  information  that  is  needed  to 
develop  cost-effectiveness  analyses.  The  quality  of  the  information  provided  is  not 
sufficient  to  provide  detailed  policy  guidance,  but  docs  appear  to  offer  some  preliminary 
insights. 

OPTEMPO  Versus  Flight  Simulators 

Prior  studies  of  the  cost  of  training  in  actual  equipment  versus  flight  simulators  in 
all  services  show  that  representative  simulator  operating  costs  vary  in  a  wide  range  around 
10%  of  actual  equipment  operating  costs  per  hour  trained.  Acquisition  costs  for  simulators 
also  have  large  variation,  and  can  be  as  high  as  actual  equipment  costs.  Fragmentary 
information  on  break-even  costs  suggest  that  representative  simulators  can  be  amortized  in 
approximately  two  years.  The  conclusions  draw’ii  from  these  studies  arc  that  the  relative 
cost  of  simulators  is  10%  of  actual  equipment  when  both  actual  equipment  and  simulators 
are  already  in  the  inventoiy,  but  closer  to  33%  when  simulators  must  be  both  acquired  and 
maintained.  Recent  information  on  seleaed  systems  arc  at  least  as  low  as  these  estimates. 

With  respect  to  the  relative  effectiveness  of  simulators  and  actual  equipment,  TER 
comparisons  from  the  literature  suggest  that  the  majority  (59%)  of  tasks  trained  have  TERs 
greater  than  0.33,  and  94%  have  TERs  greater  than  0.1. 

Combining  this  cost  and  effectiveness  information  means  that  individuals  can  be 
cost-effcctivcly  trained  on  most  tasks  using  simulators  at  current  ratios  of  simulator  to 
actual  equipment  training.  In  addition,  there  arc  tasks  that,  for  safety,  environmental,  or 
security  reasons,  are  not  cuirently  trained  using  actual  equipment  Many  of  these  tasks  can 
be  trained  using  simulators. 

Based  on  this  evidence  regarding  relative  training  costs  and  effectiveness,  it  would 
not  be  overly  risky  to  transfer  a  small  proportion'(pcrhaps  5%)  of  the  flying-hour  budget  to 
simulator  acquisition  and  operation.  Some  additional  funding  might  be  needed  initially  to 
maintain  readiness  while  simulators  arc  acquired.  After  a  period  of  one  to  two  years  of 
investing  these  transferred  funds,  the  results  could  be  used  to  cither  cost-effectively 
increase  overall  readiness  or  to  maintain  a  given  level  of  readiness  at  a  lower  overall  cost. 
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The  data  warn  against  considering  large  transfers  of  this  type  without  further 
analysis.  The  savings  from  a  5%  decrease  in  flying  hours  could  be  used  to  generate  a  15% 
to  50%  percent  increase  in  simulator  hours  (based  on  the  10%  to  33%  cost  range  discussed 
above).  The  effects  of  those  sorts  of  increases  on  TERs  should  be  understood  before  still 
greater  simulator  utilization  is  undertaken,  and  we  believe  experiments  should  be 
undertaken  to  develop  this  understanding. 

SIMNET,  NTC,  and  Home-Station  Exercises 

We  found  effectiveness  comparisons  between  SIMNET  and  home-station  field 
exercises  and  a  comparison  of  the  incremental  effect  of  home-station  field  exercises  on 
NTC  performance.  These  comparisons  are  insufficient  to  support  conclusive  findings. 

Analysis  of  an  effectiveness  comparison  between  SIMNET  and  home-station  field 
training  supports  the  hypothesis  that  SIMNET  is  extremely  effective  in  increasing 
performance  for  SIMb'ET-trainable  tasks  relative  to  standard  field  training. 

Tradeoff  analyses  were  done  between  vehicle  OPTEMPO  and  networked  simulator 
costs.  They  showed  that  investment  now  in  SIMNET-like  facilities  could  be  repaid  once 
the  facilities  were  fielded  by  a  decrease  of  8%-14%  of  OPTEMPO.  The  14%  figure  is 
similar  to  an  Army  estimate  of  the  vehicle  mileage  that  would  be  substituted  for  by  the 
SIMNET  follow-on,  the  Close  Combat  Tactical  Trainer  (CCTT). 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  section  of  the  paper  briefly  reviews  the  conclusions  from  the  data  and  analyses 
section,  and  then  presents  recommendations  for  further  research.  The  purpose  of  the 
research  is  to  supply  decision  makers  with  further  information  on  the  cost-effectiveness  of 
incremental  changes  in  simulator  use. 

Individual  Simulators  for  Training 

The  cost-effectiveness  of  simulators  for  individual  operator  and  maintainer  training 
has  been  studied,  and  there  is  a  body  of  literature  in  this  area.  However,  this  issue  needs 
additional  experimental  research  for  the  following  reasons: 

•  The  cost  of  simulators  is  heavily  based  on  digital  electronic  technology,  and  that 
cost  will  continue  to  fall  in  the  future. 

•  Political  and  environmental  changes  are  likely  to  further  constrain  the  use  of 
actual  equipment  and  field  exercises  versus  operator  and  maintainer  simulators. 
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•  Research  indicates  that  the  marginal  impact  of  additional  simulator  use  on 
performance  can  change  dramatically  depending  on  the  amount  of  simulator 
training. 

Networked  Simulators  for  Collective  Training 

Each  training  approach:  home-station  field  training,  networked  simulation,  and 
NTC,  is  cost-effective  in  training  some  tasks.  Only  extreme  changes  in  the  cost  or 
effectiveness  of  a  given  option  would  alter  this.  However,  if  the  findings  summarized 
previously  are  reinforced  by  the  experiments  suggested  here,  SIMNET-like  facilities  should 
be  developed  and  substituted  for  some  home-station  field  exercises,  and  the  disUnbution  of 
training  tasks  among  these  locations  should  be  re-examined  in  order  to  better  utilize  the 
strengths  of  each. 

Experiments  should  be  developed  to  examine  the  relative  contributions  to  combat 
effectiveness  of  NTC,  field  exercise,  and  networked  simulator  facilities.  Existing  data  do 
not  fully  support  the  sort  of  analyses  that  are  needed.  Without  additional  information, 
resource  allocation  decisions  will  continue  to  be  made  without  adequate  knowledge  of  their 
implications.  Where  possible,  these  analyses  should  generate  data  on  the  effects  of 
incremental  changes  in  simulator  and  actual  exercise  training  so  that  policy  makers  are 
better  informed  about  the  cost-effectiveness  tradeoffs. 

In  addition,  networked  simulator  cost-effectiveness  issues  could  be  addressed 
empirically  by  extending  recent  analysis  of  the  effects  of  home-station  field  exercise  activity 
on  NTC  performance  to  include  SIMNET  activity.  By  including  both  home-station  field 
exercises  and  SIMNET-like  facilities  in  analysis  of  NTC  performance,  we  should  be  able  to 
better  understand  the  process  of  producing  readiness  or  warfighting  capability  through  all 
collective  amning  options. 
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I.  INTRODUCTION 


This  report  addresses  the  issue  of  whether  increased  use  of  simulators  and  training 
devices  for  training  military  personnel  is  cost-effective.  Information  is  included  on 
simulators  used  by  both  individuals  and  crews,  recent  advances  in  networked  simulators, 
and  wargaming  technologies.  Individual  and  crew  simulators  provide  individual  operator 
and  raaintainer  training.  Networked  simulators  can  be  used  to  provide  collective 
“warfighting”  U’aining  for  relevant  Department  of  Defense  (DoD)  elements  such  as  units  in 
(a)  the  individual  military  services,  (b)  joint  commands  consisting  of  units  from  two  or 
more  individual  services,  and  (c)  combined  commands  that  include  allied  forces  as  well  as 
U.S,  joint  commands. 

The  analysis  is  presented  in  four  sections,  each  of  which  addresses  specific 
questions,  as  follows; 

•  Basic  Issues.  Is  there  a  mismatch  between  the  measured  cost-effectiveness  of 
various  approaches  to  “warfighting"  training  and  their  actual  application?  To 
what  extent  are  changing  costs  (e.g.,  the  fall  in  the  price  of  digital  electronics 
used  in  simulations,  simulators,  and  training  devices,  and  the  rise  in  the  cost  of 
actual  equipment)  and  emerging  technologies  (e.g.,  the  networking  of 
simulators  and  other  training  options)  likely  to  change  the  cost-effective 
balance  of  training  options?  How  should  we  measure  this  mismatch  and  these 
trends? 

•  Current  Policies  and  Current  Practices.  What  are  the  operative  current  policy 
documents?  To  what  extent  do  they  address  the  basic  issues?  What  are  the 

I  current  practices  in  the  uses  of  simulators  and  training  devices  for  training? 

•  :  Data  and  Analyses.  What  sort  of  data  are  available  to  allow  us  to  understand 
I  the  effectiveness  and  cost  of  various  training  options?  What  evidence  is  there 
1  that  technology  can  be  used  (or  better  used)  to  improve  the  current  practices? 
i  Can  we  use  this  evidence  to  generate  first-order  cost-effectiveness  analyses  of 
\  simulations  compared  with  other  training  options? 

•  Conclusions  and  Recommendations.  What  conclusions  can  be  drawn  from 
the  data  and  analyses?  How  should  training  practice  be  changed  on  the  basis 
of  existing  evidence?  In  what  areas  might  further  analyses — including  further 
original  or  secondary  research  efforts — be  warranted?  If  such  further  analyses 
or  research  and  development  were  conducted  and  successfully  completed,  what 
differences  would  they  make,  and  what  would  the  differences  be  worth? 
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Keeping  the  scope  of  this  paper  to  individual  and  networked  simulators  and  training 
devices  is  difficult.  Most  peacetime  military  activity  can  be  looked  on  as  preparation  for 
warfighting,  and  much  of  that  preparation  is  simulation  in  a  general  sense.  Additionally, 
technological  and  operational  realities  are  blurring  distinctions  and  allowing  interoperability 
among  simulators,  war  games  and  warfare  models,  equipment  stimulators,  insuiimented 
and  uninstrumented  ranges,  and  even  mission  rehearsal  The  latter  point  means  that, 
throughout  this  paper,  we  will  tend  to  use  the  term  “simulators”  to  represent  the  continuum 
of  training  equipment  options  that  stops  short  of  actual  operations. 
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II.  BASIC  ISSUES 


This  section  provides  a  brief  statement  of  the  overarching  goal  of  tmining  and  a 
taxonomy  that  describes  the  current  organization/location  approach  to  delivering  this 
training.  Then,  because  of  this  paper’s  focus  on  simulation  as  a  training  method,  examples 
of  the  history  and  effectiveness  of  individual  and  collective  warfighting  simulations  are 
provided. 

A.  GOAL  OF  TRAINING 

The  goal  of  training  is  to  create  a  military  that  is  “capable  of  carrying  out  national 
security  missions  in  peace  and  war  as  called  upon”  ([1],  p.  8).  In  order  to  accomplish  this 
goal  in  a  cost-effective  manner — where  cost  includes  lives  as  well  as  dollars — individuals 
must  have  mastered  the  skUIs  and  knowledge  to  perform  the  tasks  that  make  up  their 
individual  jobs,  and  these  individuals  must  be  trained  as  part  of  collectives  (from  fine  teams 
and  air  crews  to  combined  commands).  Differences  between  individual  and  collective 
training  requirements  result  in  different  design  and  delivery  approaches  to  training. 

B  .  A  TRAINING  TAXONOMY 

Military  training  can  be  categorized  into  a  two-by-two  matrix  as  shown  in  Figure  1. 
This  taxonomy  divides  training  by  environment  (school  versus  unit)  and  by  focus 
^Individual  versus  collective).  For  each  cell  in  the  matrix,  we  will  discuss  whether: 
(a)  training  received  is  well-quantified  or  not;  (b)  it  is  easy  or  difficult  to  quantitatively 
describe  training  requirements  and  compare  training  effectiveness;  and  (c)  simulations  and 
simulators  are  or  could  be  used. 

Most  of  ihc  formally  delivered  military  training  to  which  DoD  policies  apply  is 
conducted  in  military  schools  or  institutional  environments  and  focuses  on  tiie  individual 
soldier,  sailor,  marine,  or  airman.  Much  of  it  is  technical  or  specialty  training  designed  to 
provide  individual  trainees  with  the  knowledge  and  skill  essential  to  the  performance  of 
their  assigned  jobs.  This  sort  of  training  is  typically  associated  with  fixed  facilities. 
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faculties,  curricula,  and  procedures  that  are  essentially  similar  to  those  of  civilian  school 
systems.* 
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Note:  Adapted  from  Reference  [2]. 

Figure  1.  A  Taxonomy  of  Military  Training 


Very  little  collective  training  is  provided  in  schools,  and  most  of  that  is  more 
propterly  classified  as  professional  education,  e.g.,  senior  service  and  defense  colleges  and 
universities.  Such  professional  education  is  outside  the  scope  of  this  paper,  and  is  not 
considered  further. 


Unlike  the  facilities  and  equipment  u.sed  for  individual  training  in  schools,  the 
facilities  and  equipment  in  units  may  be  poorly  suited  to  the  desired  collective  training.  The 
unit’s  leaders  have  to  manage  the  training  with  a  “faculty”  drawn  from  the  unit’s  assigned 
personnel.  Some  of  these  personnel  are  not  much  more  experienced  in  the  subject  matter 
than  are  their  students,  and  most  have  not  been  trained  in  instructional  techniques.  The 
curriculum  is  constantly  changing  and  is  complicated  by  the  availability  of  resources  such 
as  ranges,  and  on-going  requirements  for  individual  on-the-job  training  delivered  by  first- 


For  additional  analysis  of  the  cost-effectiveness  of  current  individual  training  delivery  systems,  see 
Reference  [3]. 
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line  supervisors — training  that  is  largely  unstructured  and  unevaluated,  and  often  with  only 
poor  records  of  skill  acquisition  and  maintenance.  Actual  operations  enhance  some  training 
activities,  but  operational  priorities  further  fragment  others. 

While  individual  training,  particularly  at  school,  is  essential  to  give  personnel  the 
ability  to  perform  their  jobs,  most  of  the  needed  “wanlghting”  training  can  be  met  only  by 
training  of  the  “collectives” — the  crews,  groups,  teams,  or  units  (CGTUs) — that  are 
combined  to  constitute  a  military  unit,  e.g.,  an  Army  or  Marine  platoon,  company,  or 
battalion,  a  Navy  ship,  battle  group,  or  battle  force,  and  an  Air  Force  (or  Army,  Navy,  or 
Marine  Air)  flight,  squadron,  or  wing.  Such  collective  unit  training  is  directed  towards  the 
development  of  teamwork  among  the  individuals  within  a  CGTU — and  among  the  CGTUs 
within  a  larger  military  unit — who  are  to  perform  (often  stressful)  tasks  and  missions  in 
common.  It  is  exactly  the  type  of  training  central  to  the  preparation  of  a  military  unit  for 
battle. 

Collective  unit  training  has  been  less  intensively  studied  than  schoolhouse  training 
(see  Reference  [4]).  This  is  partially  because  it  is  often  difficult  to  determine  whether  such 
training  is  actually  occurring;  that  is,  whether  combat-relevant  skills  are  being  correctly 
learned,  executed,  and  maintained.  Because  of  this,  it  is  difficult  to  determine  what 
resources  were  used  for  training.  Also,  it  is  difficult  to  develop  meaningful  measures  of 
effectiveness  (MOEs)  for  many  collective  unit  training  activities  when  the  immediate 
feedback  of  casualties  suffered  and  inflicted  are  unavailable.  In  summary,  measuring  the 
performance  of  many  different  collectives  networked  together  has  been  extremely  difficult 

This  comparison  highlights  the  difficulties  in  providing  warfighting  training  in 
general,  and  collective  unit  warfighting  training  in  particular.  One  approach  to  overcoming 
them  is  to  simulate  a  combat  environment  in  order  to  train  part  of  what  can  be  learned 
through  actual  combat. 

C.  HISTORY  OF  SIMULATING  WARFARE  AS  TRAINING 

The  concept  of  developing  a  “practice  war”  as  a  method  of  training  a  force  for  actual 
combat  may  be  as  old  as  war  itself.  Models,  simulations,  and  war  games  to  represent 
warfare  are  said  to  predate  recorded  history.  Indeed,  the  use  of  elementary  forms,  such  as 
iconic  figures,  sand  tables,  chess-like  game  boards,  and  field  exercises  or  other  operations, 
has  been  documented  from  pre-Roman  times.  However,  there  appears  to  have  been  little 
formalization  of  the  procedures  for  more  abstract  simulations  until  professional  military 
colleges  and  schools  were  established  in  the  18th  century.  The  appearance  of  the  German 
“Kriegspiel”  in  181 1  marked  the  beginning  of  the  use  of  large  and  complex  manual  games 


that  employed  maps  and  rules  based  on  actual  operational  doctrine.  From  the  Franco- 
Prussian  War  throughout  World  War  II,  every  major  military  power  relied  heavily  on  the 
use  of  institutionalized  manual  war  games  for  force  planning,  training,  doctrine 
development,  and  evaluation  of  war  plans.  Since  then,  the  manual  games  have  been  largely 
replaced  by  computer-based  models,  simulations  and  war  games.  By  1986,  the  Joint 
Chiefs  of  Staffs  “Catalog  of  War  Gaming  and  Military  Simulation  Models”  listed 
approximately  600  different  active  simulations  [5],  up  from  360  in  1982. 

D.  EFFECTIVENESS  OF  EXERCISES  IN  TRAINING  FOR  COMBAT 

This  section  presents  evidence  that  there  are  vast  differences  in  the  combat 
performance  of  individuals  and  collectives,  that  combat  experience  is  a  primary  determinant 
of  combat  performance,  and  that  simulating  combat  can  partially  substitute  for  actual 
combat. 

There  is  some  evidence,  and  a  considerable  number  of  testimonials  from  warriors 
and  military  historians,  of  vast  differences  in  the  battle  effectiveness  of  commanders  and 
their  units — much  of  these  attributable  to  differences  in  experience  and  collective  training. 
Naval  commanders  and  historians  know  of  “good  ships”  and  “poor  ships,”  often 
representing  essentially  identical  hardware  in  ships  of  the  same  class.  Army  commanders 
and  historians  know  of  “good  divisions”  and  “poor  divisions,”  identically  equipped  and 
often  fighting  side-by-side. 

Data  are  available  to  support  these  assumptions.  For  example,  one  NATO- 
sponsored  study  of  World  War  11  German  U-boat  commanders  found  that  32.9%  failed  to 
engage  any  targets,  and  13.1%  failed  to  hit  any  targets  they  found.  Thus,  46%  were 
ineffective.  On  the  other  hand,  the  best  10%  of  the  commanders  sank  45%  of  the  Allied 
ships  sunk  by  submarines  [6]. 

Another  example  is  drawn  from  the  World  War  II  air-battle  experiences  of  the 
German  and  American  forces  in  the  European  theater.  This  is  shown  by  the  air-combat 
loss  rates  represented  in  Figure  2.  The  German  pilots  experienced  nearly  40%  first- 
mission  losses,  whereas  the  American  first-mission  loss  rate  was  half  that,  or  about  20%. 
The  figure  shows  essentially  a  five-mission  advantage  for  the  American  pilots  relative  to  the 
German  pilots.  This  advantage  has  been  attributed  to  the  differences  in  their  training.  The 
Germans  were  short  of  fuel,  and  although  they  trained  their  pilots  well  to  fly  the  aircraft, 
they  did  not  have  fuel  to  “waste”  on  the  additional  training  of  air-combat  maneuvering  skills 
[2  and  7]. 


Figure  2.  Air  Combat  Loss  Ratas  in  World  War  li 

Another  example  of  differences  that  result  from  training  through  combat  simulation 
is  shown  in  the  exchange  ratios  for  the  Vietnam  air  war  that  arc  shown  in  Table  1.  The 
table  shows  aircraft  exchange  ratios  from  two  4-year  time  periods,  where  the  omitted  year 
had  no  losses.  The  exchange  ratios  shown  are  an  overall  ratio,  followed  by  ratios  for  the 
U.S.  Air  Force  (USAF)  and  the  U.S.  Navy  (USN).  'fhe  data  to  construct  these  ratios  are 
then  presented. 

The  major  differem  .  in  the  data  is  the  5:1  imprx)vement  in  the  USN  exchange  ratio 
for  the  later  period.  The  difference  between  these  periods  is  the  establishment  of  the  Top 
Gun  Fighter  Weapons  School,  a  school  that  used  engagement  simulation  (mock  combat) 
and  intensive  after-action  reviews  of  the  simulated  combat  to  train  pilots  [2]. 
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Table  1. 


'\  ■  \ 


:hange  Ratios  In  the  Vietnam  Air  War 


Time  Period 

Cateuorv 

1965-1968 

1970-1973 

Overall  Exchange  Ratio 

2.29 

2.74 

USAF  Exchange  Ratio 

2.25 

2.00 

USN  Exchange  Rado 

2.42 

12.50 

Enemy  Losses 

no 

74 

US  Losses 

48 

27 

USAF  Kills 

81 

49 

USAF  Losses 

36 

25 

USN  Kills 

29 

25 

USN  Losses 

12 

2 

ource;  Reference  [2]. 

E.  OPERATIONS,  EXERCISES,  SIMULATORS,  AND  COST- 
EFFECTIVENESS 

This  section  discusses  different  ways  that  simulators  can  be  used  to  augment 
exercises  or  operations  as  warfare  simulations.  It  then  offers  a  conceptual  approach  to 
analyzing  the  cost-effectiveness  of  these  simulators.  Hnally,  several  other  issues  that  affea 
the  cost-effectiveness  of  simulators  are  discussed. 


Simulations  may  allow  training  through  representation  of  a  type  of  actual  operation 
or  exercise  that  is  effectively  precluded  by  political,  safety,  security  or  budget  constraints, 
or  it  may  allow  a  type  of  training  that  substitutes  for  a  particular  operation.  Simulations 
may  substitute  directly  (a  particular  task  or  maneuver  can  be  trained  almost  entirely  through 
simulation)  or  indirectly  (training  seme  tasks  through  simulation  leaves  additional  operating 
time  for  training  actual  tasks  that  are  not  easily  learned  through  simulation).  In  addition, 
using  simulators  for  direct  mission  rehearsal  for  specific  operations  is  discussed  in  section 
n.E.5,  Other  Factors  that  Affect  Simulator  Gist-Effectiveness. 


1 .  Training  for  What  Could  Otherwise  Not  Be  Done 

One  example  of  actual  training  that  is  pi^luded  by  political  constraints  is  to  practice 
hostile  activities  on  a  potential  adversary’s  territory  to  understand  the  unique  characteristics 
of  the  operation  based  on  that  territory.  An  iuteroative  approach  is  to  conduct  actual 
exercises  over  similar  terrain  similar  to  that  of  the  adversary  and  to  which  access  is 
available.  Another  is  to  use  a  simulator  that  can  project  a  realistic  representation  of  the  area 
to  be  attacked,  such  as  simulated  bombing  runs  on  Moscow  or  Baghdad. 
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An  example  of  allied  or  domestic  political  constraints  would  be  the  unwillingness  to 
have  large,  intrusive  exercises  occur  frequently  over  particular  territory.  Specific  examples 
of  this  constraint  might  include  the  Federal  Republic  of  Germany’s  reluctance  to  allow 
frequent  NATO  exercises  over  its  territory.  Another  example  is  that  even  if  Washington, 
D.C.,  were  a  perfect  match  for  a  major  potential  target  for  low-level  B-52  attacks,  political 
considerations  would  make  it  inappropriate  to  practice  such  attacks. 

Simulators  could  be  substituted  for  either  operational  or  maintenance  training  when 
the  activities  in  question  are  too  dangerous  or  destructive  to  be  uained  on  actual  equipment, 
or  where  frequent  use  of  the  equipment  would  offer  the  potential  for  security  compromises 
with  respect  to  operating  characteristics  or  procedures. 

Determining  the  co.st-effectiveness  of  operational  training  has  been  difficult  because 
even  when  the  training  was  possible,  both  the  costs  and  benefits  were  difficult  to  quantify. 
However,  recently  developed  networked  simulator  technologies  make  some  of  these 
simulations  pos.cible,  and  others  allow  the  costs  and  benefits  of  some  simulations  to  be 
better  measured.  Specifics  of  this  approac  h  will  be  discussed  in  the  following  sections. 

2.  Simulators  as  Substitutes  for  Acvual  Equipment 

Even  when  actual  equipment  can  be  used  in  the  field  for  collective  training,  there 
may  be  reasons  to  use  simulators  instead.  Training  using  simulators  allow  for  small 
incremental  changes  to  the  balance  of  simulation  and  actual  operation.  For  example,  if  the 
tasks  of  a  particular  platform  operator  can  be  separately  identified  and  trained,  then  tasks 
where  simulated  training  is  particularly  cost-effective  could  be  trained  almost  exclusively 
through  simulation.  This  would  result  in  either  a  budgetary  savings,  or  the  opportunity  to 
undertake  additional  training  on  actual  equipment  to  train  difficult  tasks  that  are  not  well 
simulated. 

An  additional  condition  that  can  favor  simulation  versus  actual  equipment  training  is 
range  availability.  For  example,  according  to  the  Government  Accounting  Office  (GAO), 
one  conclusion  of  a  Navy  training  assessment  was  that  “there  are  limited  numbers  of 
ranges,  most  of  which  lack  the  ability  to  provide  a  realistic  threat  environment  for  training” 
([8],  p,  15).  Limitations  on  ranges  may  result  from  demographic  changes  (surrounding 
areas  that  were  once  uninhabited  are  now  populated)  or  technological  changes  (the  tactical 
reach  of  new  weapon  systems  is  becoming  greater).  In  addition,  this  G.\0  report  cites 
earlier  GAO  work  that  found  shortages  in  military  training  airspace  “decreased  aircrew 
training  effectiveness,  caused  some  units  to  deploy  significant  distances  to  areas  where 
airspace  was  available,  and  caused  some  units  to  obtain  waivers  for  some  training 
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requirements  to  avoid  reporting  degraded  readiness”  ([8],  p.l6).  Similar  comments  were 
made  by  Army  helicopter  training  instructors,  who  pointed  out  that  airspace  and  budgetary 
restrictions  mean  that  the  vase  majority  of  training  that  pilots  receive  in  the  employment  of 
certain  advanced  missiles  is  through  simulator  use  [9]. 

Problems  such  as  range  availability  can  either  be  modeled  as  additional  constraints, 
or  through  a  difficult-to-quantify  increase  in  the  cost  of  training  with  actual  equipment.  In 
any  case,  this  problem  suggests  that  employing  more  simulation  has  the  potential  to  be  a 
cost-effective  strategy. 

3.  Cost-Effectiveness  of  Simulations 

This  section  provides  both  a  qualitative  and  a  quantitative  method  for  a.ssessing  the 
•cost-effectiveness  of  simulations. 

a.  Qualitative  Issues 

A  qualitative,  first-order  approximation  of  the  effectiveness  of  a  simulator  used  for 
training  is  the  realism  or  fidelity  with  which  a  simulator  reproduces  actual  equipment  and  its 
operation.  In  most  cases,  the  more  realistic  is  a  simulation  of  the  training-relevant  aspects, 
the  better  it  trains,  and  the  more  it  costs. 

However,  this  mental  model  must  be  used  carefully  for  two  reasons.  The  first 
reason  is  that,  as  opposed  to  actual  operations  with  real  equipment,  it  is  possible  to  build 
simulators  to  train  specific  tasks.  For  example,  pre-flight  cockpit  checks  can  be  trained  in  a 
simulator  without  expensive  subsystems  such  as  terrain  display,  six-axis  motion,  or  the 
airframe,  engine(s),  and  avionics  of  an  actual  airplane.  Therefore,  it  is  important  to  be  clear 
on  what  aspects  of  actual  equipment  or  operations  must  be  realistic  for  the  particular  tasks 
to  be  trained. 

Secondly,  the  relation  of  cost  to  realism  is  neither  linear  nor  the  same  for  different 
tasks.  Figure  3  presents  several  possible  relationships.  Line  A  might  represent  the 
realism-cost  relationship  for  the  pre-flight  cockpit  checks  described  above,  or  for  a  part- 
task  maintenance  simulator.  Curve  B  represents  the  concept  that  much  of  the  realism  in  a 
given  simulation  can  be  achieved  with  relatively  little  cost,  but  that  approaching  perfect 
fidelity  becomes  expensive.  An  example  of  this  is  the  often  voiced  subjective  assessment 
that  SIMNET  achieves  60%  of  the  fidelity  or  realism  of  real  equipment  at  10%  of  the  cost. 
Curve  C  might  represent  training  flying  tasks  where  realistic  visual  and/or  motion  cues  arc 
needed  such  that  the  simulator  must  have  the  (expensive)  equipment  needed  to  provide 
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these  cues.  For  an  example  of  tlie  effectiveness  change  suggested  by  discontinuous  curves 
similar  to  C,  see  Reference  [10]. 


A 


Figure  3. 


Simulator  Cost  Versus  Realism 


This  qualitative  approach  has  been  criticiad  and  refined  in  several  different  ways,^ 
It  has  been  stylized  as  “objective  reality,”  the  degree  to  which  the  simulator  replicates 
aircraft  states  and  conditions,  and  contrasted  witl^  “perceptual  fidelity,”  the  degree  to  which 
the  trainee  perceives  the  simulator  to  duplicate!  the  aircraft.  Another  concept  is  “error 
fidelity,”  which  argues  that  if  the  errors  made  iin  the  simulator  have  the  same  effects  on 
aircraft  performance,  and  the  errors  are  made  in  the  same  way  and  as  frequently  as  in  an 
actual  aircraft,  then  the  simulator  will  be  a  bettei"  trainer.  The  literature  has  generated  the 
following  definition  of  fidelity:  “Simulator  fidelity  is  the  degree  to  which  characteristics  of 
perceivable  states  induce  correct  psychomotor  and  cognitive  control  strategy  for  a  given 
task  and  environment”  [12]. 


In  many  ways,  the  perceptual,  error,  and  simulator  definitions  above  may  be 
construed  as  better  defined,  necessary  conditions  that  are  not  quite  as  restrictive  as  the  less 
well-defined,  presumably  sufficient  condition  of  realism  suggested  at  the  start  of  this 
section.  To  some  extent,  these  words  are  different  ways  of  discussing  the  concepts 
represented  in  Figure  3. 

Another  way  of  highlighting  this  issue  is  to  say  that  the  real  question  is  not  high  or 
low  fidelity,  but  what  fidelity  is  needed  to  satisfy  what  training  objective.  In  fact,  in  certain 


Ihis  discussion  draws  on  Reference  [1 1]  and  the  works  cited  there. 
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instances  the  assumption  that  greater  fidelity  means  belter  training  may  not  always  be 
correct  For  example,  in  a  simulator,  certain  meaningful  cues  for  a  particular  task  Cv/uld  be 
temporarily  enhanced  and  distracting  cues  suppressed  to  facilitate  training.  This  would 
allow  the  trainee  to  more  rapidly  identify  llie  important  elements  of  particular  taslts.  ,4s  the 
trainee  became  familiar  with  the  needed  response,  the  intensity  of  the  cues  could  be 
changed. 

b.  Quantitative  Issues 

A  well-known  and  frequently  used  quantitative  method  for  comparing  simulated 
versus  actual  operation  is  the  so-called  training  Transfer  Effectiveness  Ratio  (TER),  The 
TER  was  described  in  the  training  literature  [13  and  14],  It  is  a  ratio  of  actual  equipment 
training  time  saved  as  a  function  of  time  spent  training  on  a  simulation.  An  algebraic 
representation  of  this  ratio  is  (Awo  -  Aw)/S,  where  A^o  is  the  training  time  in  the  actual 
equipment  or  situation  without  (wo)  use  of  a  simulator.  Aw  is  the  actual  time  after  (with 
(w))  simulator  training,  and  S  is  the  (non-zero)  time  spent  in  the  simulator.  This  ratio  can 
be  used  for  total  or  cumulative  time  spent  in  simulators  and  actual  equipment,  or  for 
incremental  changes  in  simulator  time  that  result  in  incremental  changes  in  actual 
equipment  Large  values  of  this  ratio  indicate  that  simulations  train  relatively  well  in 
comparison  to  operating  actual  equipment,  and  small  values  indicate  that  simulations  train 
poody  relative  to  actual  equipment  There  is  also  empirical  evidence  that  this  ratio  declines 
as  the  value  of  S  increases,  e.g„  Reference  [15]. 

When  defined  as  incremental  improvements  in  capability,  the  TER  has  an 
inteipretation  similar  to  the  ratio  of  marginal  products  in  the  optimality  condition  for  cost- 
effective  use  of  inputs  in  a  production  proces-s,  (MPs/MPa)  =  (Ps/Pa).  where  MP  is 
marginal  product  and  P  is  price,  and  the  subscripts  S  and  A  are  simulator  and  actual 
equipment.  Therefore,  many  of  the  standard  results  of  microeconomic  theory  can  be 
applied  to  policy  decisions  on  how  much  to  train  using  operations  and  how  much  to  train 
using  simulation.  For  example,  simulators  should  be  used  to  train  tasks  for  which  the  TER 
is  greater  than  the  ratio  of  the  price  of  simulator  training  to  the  price  of  actual  equipment 
training. 


An  incremental  change  in  simulator  usage  (AS)  yields  a  change  in  the  actual  equipment  time  needed  to 
obtain  equal  proficiency  [A(Awo  -  Aw)].  AS  costs  Pg  per  unit,  so  the  cost  incurred  is  AS  x  Pg,  and 
A(Awo  -  Aw)  costs  Pa,  so  the  cost  that  could  be  avoided  is  A(Awo  -  Aw)  x  Pa-  Simulators  and 
training  devices  should  be  used  as  long  as  the  cost  incurred  to  attain  equal  proficiency  is  less  than  the 
cost  avoided,  or  up  to  the  point  where  the  two  costs  are  equal,  hence  to  the  point  where  AS  x  Pg  = 
A(Awo  -  Aw)  X  Pa.  Rearranging  terms  yields  A(Awo  -  Aw)/AS  =  Pg/PA  (=  MPg/MPA). 


I 

I 

a 

I 

I 

I 
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In  following  sections,  we  will  present  evidence  on  the  cost-effectiveness  of 
individual  operator  and  maintainer  simulations,  drawn  mainly  from  Orlansky  [  1 6]  and 
Fletcher  and  Orlansky  [17].  There  is  additional  evidence  suggesting  that  use  of  other  types 
of  combat  mission  simulators  for  training  can  provide  similar  results,  especially  when 
networked  for  collective  training.  For  example,  during  1987,  U.S.  Army  tank  platoons 
stationed  in  Europe  were  trained  with  a  network  of  SIMNET  tank  simulators.  The  skills 
they  gained  in  the  simulator-augmented  training  transferred  positively  to  the  NATO  exercise 
range,  and  the  U.S.  Army  tankers  for  the  first  time  in  a  decade  earned  the  first  prize  in 
competition  with  tankers  from  other  NATO  forces  [6  and  18]. 

4.  Evaluating  Training  Options 

Clearly,  there  is  a  wide  range  of  training  options  to  meet  the  goal  of  carrying  out 
national  security  missions.  What  combination  of  simulators  and  actual  equipment  (in  the 
case  of  operator  and  maintainer  training),  or  networked  simulators,  actual  equipment,  and 
field  exercises  (in  the  case  of  collective  training),  should  be  used  depends  on  the  relative 
effectiveness  and  cost  ratios  between  the  various  training  options.  There  is  no  a  priori 
reason  to  assume  that  any  of  these  options  is  completely  dominated  by  the  others  for  all 
trainable  tasks.  That  is,  there  will  be  some  training  tasks  that  are  most  cost-effectively 
undenaken  by  each  training  option.  Also,  there  may  well  be  times  when  temporary 
resource  constraints,  e.g.,  range  availability,  will  preclude  some  units  taking  advantage  of 
particular  training  approaches.^ 

The  analyucal  challenge  is  to  quantify  the  TERs  and  costs  for  alternative  training 
options  so  that  cost-effectiveness  tradeoff  analyses  can  be  used  to  help  decide  which 
options  will  be  used  to  train  which  tasks.^  Using  collective  training  as  an  example,  we 
generate  readiness  through  exercises  at  ranges,  maneuver  areas,  and  combat  training 
centers,  as  well  as  through  the  use  of  training  aids,  devices,  simulators,  and  simulations. 
The  exercises  involve  operating  equipment,  consuming  fuel,  and  expending  ordnance. 
Evaluations  of  mixes  of  training  options  require  measuring  both  the  inputs  and  outputs 
associated  with  unit  training. 


*  Wc  have  eschewed  a  production-theory-like  mathematical  treatment  because  simulators,  actual 
equipment,  and  collective  exercises  are  themselves  a  continuum  of  training  options,  and  because  the 
dau  on  TERs  do  not  currently  support  such  formalism.  For  standard  treatments  of  these  issues,  see 
[19]  or  [20]. 

^  For  an  example  of  one  service  perspective  on  these  issues,  see  [21]. 
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5.  Other  Factors  that  Affect  Simulator  Cost*Effectiveness 

There  are  many  other  factors  that  have  or  will  affect  the  overall  cost-effectiveness  of 
simulators.  Three  are  discussed  below.  They  are  the  falling  cost  of  digital  electronic 
hardware,  which  has  been  lowering  the  cost  of  a  given  amount  of  simulator  capability;  the 
increased  use  of  computer  software  in  simulations,  which  has  increased  capability  but  also 
costs;  and  the  use  of  training  systems  as  mission  rehearsal  devices,  which  may  result  in 
cost  sharing  of  simulator  development. 

a.  Digital  Electronics 

Digital  electronics,  in  particular  computers  and  displays,  give  some  simulators 
much  of  their  capability.  In  some  cases,  they  also  reflect  much  of  their  cost.  On  average, 
the  constant  dollar  price  of  a  given  amount  and  type  of  computer  processing  power  has 
been  falling  at  the  rate  of  25%  per  year*  (see  [22  through  24]).  In  addition,  substitution  of 
microcomputers  for  larger  computers  in  simulation  applications  bring  one-time  savings  .of 
one  to  two  orders  of  magnitude  in  cost  per  amount  of  processing  power  [25].  There  ore 
reasons  to  believe  this  decline  will  continue  through  this  decade  [26  and  27]. 

These  price  declines  have  been  used  to  both  lower  the  price  and  increase  the 
capability  of  simulations.  Also,  such  price  declines  indicate  that  the  capabilities  of  ( >lder 
simulators  could  be  purchased  less  expensively  now.  This  helps  to  make  the  ost- 
effectiveness  results  from  older  studies  biased  in  a  conservative  direction,  that  is,  against 
simulations.  For  a  fuller  discussion  of  computer  hardware  price  decreases  and  their  impact 
on  training,  see  Angier  and  Fletcher  [3]. 

b.  Computer  Software 

Because  computer  hardware  is  cheaper,  more  of  it  is  used  in  the  production  of 
platforms,  weapons  systems,  and  simulators.  To  control  this  hardware,  and  to  add 
additional  functionality,  more  and  more  complex  software  is  needed.  The  cost  of  a  given 
amount  of  functionality  implemented  in  software  has  fallen  in  the  range  of  5%  per  year  [28 
and  29]  and  it  is  hoped  that  this  decline  can  be  accelerated  [30].  However,  the  increased 
requirements  for  functions  that  must  be  implemented  through  computer  software  mean  that 
software  expenditure  will  have  to  rise  to  accomplish  these  tasks."^ 


Examples  of  types  of  computers  are  mainframes,  minicomputers,  and  microcomputers.  This  rate  of 
price  declme  is  a  little  lower  for  mainframes,  ano  a  little  higher  for  microcomputers. 

FCr  ei^ple,  intei^y  generated  data  at  IDA  suggest  annual  growth  rates  for  on-board  software  for 
several  aircraft  in  the  range  of  20%  per  year. 
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c.  Using  Training  Systems  for  Mission  Rehearsal 

An  emerging  use  for  simulators  and  simulations  is  as  a  foundation  for  mission 
rehearsal  systems.  Mission-rehearsal  systems  are  envisioned  as  particularly  valuable  in  a 
world  where  the  scenario  is  not  known  until  just  a  few  days  before  the  operation  must 
commence.  They  are  particularly  valuable  for  training  relatively  small  CGTUs. 

An  example  of  this  use  of  simulators  is  the  Special  Operations  Forces- Air  Training 
System  (SOF-ATS).  The  SOF-ATS  is  being  designed  and  developed  to  be  a  dual-purpose 
system  that  can  be  used  for  either  standard  training  or  rehearsal  for  specific  missions.  The 
system  will  network  seven  different  aircraft  types  (two  transport  aircraft,  two  gunships, 
one  tanker,  and  two  helicopters)  totaling  50  crew  positions.  A  key  requirement  for  this 
system  that  is  specific  to  mission  rehearsal  is  the  ability  to  rapidly  construct  a  terrain  image 
of  target  areas  and  flight  paths  to  and  from  the  target.  The  life-cycle  cost  is  projected  to  be 
at  least  $500  million  [31]. 

With  respect  to  training  alone,  the  cost-effectiveness  impacts  of  these  dual-use 
systems  arc  not  yet  clear.  Requirements  for  mission-rehearsal  features  may  increase  the 
cost  of  simulators.  However,  dual-use  simulators  offer  cost  sharing  of  development 
expenditures,  and  the  additional  features  may  result  in  better  training,  even  for  tasks  that 
were  previously  trained  using  simulators. 


F.  SUMMARY 


In  summary,  training  for  individuals  and  collectives  occurs  in  both  formal  and 
informal  settings.  While  training  of  individual  skills  and  knowledge  in  schools  is  a 
necessary  part  of  overall  combat  readiness,  collective  unit  training  is  central  to  the 
preparation  of  a  military  unit  for  battle.  There  is  sufficient  evidence,  both  anecdotal  and 
quantifiable,  to  indicate  that  combat  experience  is  the  best  trainer  of  combat  skills.  Other 


things  equal,  combat  veterans  have  out-performed  “green”  troops  in  every  recorded 
engagement.  Simulations  have  long  been  used  to  augment  actual  operations,  o'r  as  a 
substitute  where  actual  operations  or  field  exercises  are  not  possible,  not  feasible,  or  too 
costly.  We  now  have  both  the  technological  capability  to  create  training  options  based 
more  heavily  on  computer  simulations  and  conceptually  sound  methods  to  measur^  the 
cost-effectiveness  of  these  approaches. 


III.  CURRENT  POLICIES  AND  PRACTICES 


This  section  has  two  purposes.  The  first  is  to  describe  particular  DoD  policy- 
specifying  documents  regarding  simulation  and  training.  The  second  is  to  briefly  describe 
some  of  the  ways  simulators  are  used  for  individual  and  collective  training.  These 
discussions  are  not  designed  to  be  encyclopedic,  but  to  highlight  the  areas  of  particular 
interest  for  this  paper. 

A.  CURRENT  POLICIES 

The  two  principal  DoD  policy-specifying  documents  regarding  simulation  and 
training  are  (a)  DoD  Directive  1322.18,  Military  Training,  dated  9  January  1987  [32],  and 
(b)  DoD  Directive  1430.13,  Training  Simulators  and  Devices,  dated  22  August  1986  [33]. 
The  Assistant  Secretary  of  Defense,  (Force  Management  and  Personnel)  [ASD(FM&P)],  is 
the  cognizant  and  responsible  official  for  both  documents. 

1.  DoD  Directive  1322.18,  Military  Training 

DoD  Directive  1322.18  defines  military  training  as  “instruction  and  applied 
exercises  for  the  acquisition  and  retention  of  skills,  knowledge,  and  attitudes  required  to 
accomplish  military  tasks”  ([32],  p.  1]).  It  further  defines  collective  training  as  “instruction 
and  applied  ex  :rcises  that  prepare  an  organizational  team  (such  as  a  squad,  air  crew, 
battalion,  or  multi-service  task  force)  to  accomplish  required  military  tasks  as  a  unit”  ([32], 
p.  1).  Unit  training  is  “individual  or  collective  training  conducted  by  an  operational  unit” 
([32],  p.  1). 

The  directive  goes  on  to  assert  a  systems  approach  to  training,  pointing  out  that  all 
types  of  military  training  shall  be  considered  as  interdependent  parts  of  an  overall  training 
system.  It  states:  “AUocadon  of  resources  for  the  training  of  military  individuals  and  units, 
including  those  of  the  reserve  components,  shall  be  consistent  with  assigned  wartime 
missions  and  employment  and  deployment  schedules  and  related  requirements  for  training” 
([32],  p.  2).  It  calls  for  “vigorous  research  programs  ...  for  developing  innovative  uses 
of  training  technology  to  make  military  training  programs  more  effective  and  efficient” 
([32],  p.  2). 


17 


With  specific  regard  to  the  application  of  simulation,  DoD  Directive  1322.18  states 
([32],  p.  2): 

Simulators  and  other  training  devices  for  weapon  systems  and  equipment  shall  be 
developed,  procured,  oistributed,  and  used  when  they  are  capable  of  effectively  and 
economically  supplementing  training  on  the  actual  equipment.  Particular  emphasis  shall 
be  placed  on  simulators  that  provide  training  that  might  be  limited  by  safety 
considerations  or  constraints  on  training  space,  time,  or  other  resources.  When  deciding 
on  simulation  issues,  the  primary  consi^ration  shall  be  improving  the  quality  of  training 
and  consequently  the  state  of  readiness.  Potential  savings  in  operating  and  support  costs 
nonnally  shall  be  an  important  secondary  consideration. 

With  regard  to  collective  training,  which  the  directive  states  “shall  serve  to  achieve 
standards  of  unit  proficiency  required  to  accomplish  wartime  missions”  {[32],  p.  5-6), 
realism  is  mandated  as  follows: 

Subject  to  such  constraints  as  safety  requirements  and  limits  on  space  for  training,  all 
collective  training  shall  be  conducted  under  conditions  and  rates  of  activity  closely 
approximating  those  that  the  units  being  trained  may  encounter  in  combat. 

(1)  When  constraints  limit  the  use  of  realistic  training  conditions,  then  simulation  and 
other  products  of  training  technology  shall  be  used  as  ^plicable  to  enhance  realism. 

(2)  Collective  training,  to  the  degree  feasible,  shall  include  elecuonic  warfare  activity; 
nuclear,  biological,  and  chemical  defense  activity;  and  the  periodic  use  of  opposing  forces 
trained  in  the  tactics  of  potential  adversaries. 

(3)  All  collective  training  exercises  shall  emphasize  realistic  performance  of  the 
fimetions  of  individual  personnel  in  the  exercising  units. 

(4)  Support  units  shall  be  integrated  into  exercises  for  realistic  training  in  their  wartime 
supporting  roles. 

For  puqjoses  of  this  paper,  it  should  be  noted  that  the  directive  calls  for  joint  and 
combined  exercises,  but  is  silent  with  regard  to  the  possibility  of  using  networks  of 
simulators  and  war  games  for  such  training.  It  does  state  that,  ‘To  the  extent  feasible, 
participation  in  operational  missions  shall  be  used  to  meet  the  collective  training 
requirements  of  the  units  involved,”  and  that,  “All  collective  training  and  exercises  shall  be 
evaluated  against  established  standards  of  mission  proficiency  for  identifying  and 
correcting  deficiencies”  ([32],  p.  6). 


2.  DoD  Directive  1430.13,  Training  Simulators  and  Devices 

DoD  Directive  1430.13  is  intended  to  establish  “training  simulator  and  device 
development,  acquisition,  and  utilization  policy”  and  to  provide  guidance  “for  establishing 
Service  policy  for  training  simulators  and  devices”  ([33],  p.  1)  and  it  emphasizes  the 
integration  of  simulators  and  training  devices  with  overall  training  systems.  It  establishes 


ASD(FM&P)  as  the  principal  (and,  indeed,  the  sole)  responsible  official  in  the  Office  of  the 
Secretary  of  Defense  (OSD)  in  matters  bearing  on  training  simulators  and  devices. 

The  directive  states  that  policy  for  simulators  and  training  devices  grows  out  of  the 
general  policy  of  optimizing  the  operational  readiness  of  the  total  force.  It  also  states  that 
development  and  acquisition  of  simulators  and  training  devices  within  the  services  shall  be 
based  on  each  service’s  training  requirements  analysis  process,  and  shall  include  a  cost- 
benefit  analysis  of  alternatives  and  their  potential  effects  on  active,  reserve  component,  and 
inter-service  training. 

Under  acquisition  guidelines,  the  directive  states  that  alternatives  to  simulators  and 
training  devices  as  well  as  alternative  simulators  and  training  devices  shall  be  evaluated  by 
the  service  concerned.  This  evaluation  should  include  life-cycle  costs  and  benefits,  and  the 
flexibility  of  the  system  in  response  to  changes  in  the  system  to  be  maintained,  and  the 
training  location,  method,  or  load.  Finally,  the  directive  mandates  that  commerciil 

j 

practices,  equipment,  and  software  may  be  used  when  military-specific  requirements  do  not 
exist 

like  DoD  Directive  1322,18,  previously  discussed,  DoD  Directive  1430.13  is  silent 
with  regard  to  the  use  of  networks  of  simulators  and  war  games  for  training.  I 

i 

3.  Summary  for  Current  Policies  ! 

I 

The  directives  develop  a  taxonomy  that  allows  us  to  organize  our  understanding  of 
training  location  and  methods.  The  directives  mandate  a  systems  approach  to  training, 
placing  all  different  sorts  of  training  within  the  overall  training  system.  They  also  suggest 
that,  particularly  for  simulators  and  training  devices,  the  primary  consideration  shall  Be 
improved  training  quality  leading  to  enhanced  readiness;  cost  issues  are  described  as  an 
important  secondary  consideration. 

The  directives  are  explicit  about  individual  and  crew  simulators  that  have  a  long 
histoiy’  in  military  training.  They  are  not  explicit  about  the  use  of  networiced  simulators  and 
war  games  in  such  training. 

B.  CURRENT  PRACTICES 

As  indicated  earlier,  collective  (or  crew,  group,  team,  and  unit — CGTU)  training, 
unlike  individual  training,  is  embedded  technically  and  financially  in  the  operating  forces. 
It  is,  in  fact,  the  operations  training  of  service  forces.  Depending  on  the  cuxurnstances. 
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what  the  forces  do  may  be  defined  as  either  training  or  operating.  Certainly,  with  proper 
feedback  and  reinforcement,  members  of  the  force  should  be  learning  while  operating. 

1 .  Collective  Training — War  Simulations  Through  Field  Exercises 

The  need  for  advances  in  collective  training  has  also  been  recognized  by  the 
services  (e.g..  Reference  [2]).  But  collective  training  in  operational  military  units,  being 
related  so  closely  to  each  service’s  operations  and  readiness,  is  regarded  as  entirely  within 
the  domain  of  the  individual  service’s  area  of  responsibility  and  authority.  Since  such 
training  is  regarded  as  fundamentally  related  to  “service-unique”  concepts,  doctrine,  and 
tactics,  each  of  the  services  has  established  its  own  collective  training  methods,  means,  and 
goals  within  its  operational  forces. 

For  example,  having  observed  a  factor-of-five  improvement  in  kill-loss  ratio  by 
Navy  pilots  in  the  Viemam  Air  War  subsequent  to  the  Navy’s  initiation  of  its  air  combat 
maneuvering  and  engagement  training  (Top  Gun),  the  Air  Force’s  Tactical  Air  Command 
established  its  version  of  such  training  (Red  Flag)  at  Nellis  Air  Force  Base  in  Nevada  “to 
enable  each  pilot  to  experience  his  first  ten  ‘decisive  combats,’  and  thereby  make  available 
to  air  commanders,  on  D  Day  of  any  future  war,  the  differential  between  40%  and  5% 
probability  of  pilot  loss  on  sortie  one,  and  the  significant  increase  in  pilots  and  aircraft  that 
would  be  available  for  sorties  two,  or  three  or  n.”  ([2],  p.  6)  The  Army’s  Training  and 
Doctrine  Command  (TRADOC)  sought,  and  later  established,  a  parallel  capability  for 
ground  forces  ([2],  p.  9, 15-16): 

Fortunately,  several  parallel  actions  were  underway  which  made  it  possible  ultimately  to 
act  afnimatively  on  TCATA’s  [i.e.,  TRADOC  Combat  Analysis  and  Test  Agency’s] 
recommendation  [that  laser-aided  engagement  simulatitm,  supplemented  by  live  fue  battle 
runs,  be  incorporated  into  armor  training  Army-wide] _ 

. . .  modem  battle  spreads  a  battalion  across  a  dauntingly  large  area,  and  therefore  .  . . 
large  amounts  of  land  are  required  to  train  it  for  war. 

At  Fort  Irwin,  California,  the  United  States  Army  had  enough  land  to  train  well  at  least 
two  battalions  of  a  brigade  in  modem  mobile  warfare — 1000  square  miles  of  uninhabited 
land. . . .  The  National  Training  Center  established  there  provides  an  intense  experience  in 
moving,  shooting,  and  communicating  lasting  some  three  weeks,  during  which 
participants  fight  eight  to  ten  “battles.”  The  training  methodology  is  centered  on 
experiential  learning,  with  emphasis  on  after-action  reviews  of  each  battle,  and  detailed 
records  of  each  battle  for  use  by  the  unit  in  its  subsequent  uaining  at  home  station.  The 
enabling  techniques  include  both  TES  (i.e.,  tactical  engagement  simulation]  and  live 
firing,  instrumentation  of  participants  which  permits  recording  maneuvers  and  firing 
events  (real  or  simulated),  and  use  of  a  resident  Opposing  Force  (OPFOR),  modeled  upon 
a  Soviet  Motorized  Rifle  Regiment.  Both  sides  are  allowed,  within  the  limits  of  safety, 
fiee  maneuver  in  their  operations,  and  neither  is  provided  any  information  of  the  other  it 
does  not  earn  through  reconnaissance.  But  OPFOR  has  the  advantage  of  knowing  the 
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hostile,  complex  terrain  very  well,  and  it  employs  Red  Army  doctrine  with  ruthless 
efficiency  to  expose  dependably  the  slightest  tactical  ineptitude  on  the  part  of  a  unit  in 
training.  Since  its  opening,  over  one  hundred  rotations  of  the  latter  have  arrived  at  Fort 
Irwin  to  give  battle  to  OPFOR,  and  only  a  few  would  dc;.c'‘be  their  experience  as 
“victorious.”  Yet  in  a  larger  sense,  the  whole  Army  has  been  a  winner,  for  the  small 
defeats  in  the  Mojave  Desert  have  spurred  officers,  noncommissioned  officers,  and  soldiers 
to  new  professional  heights. 

Understandably,  the  tactical  engagement  simulation-based  collective  training  at  Fort 
Irwin  is  very  expensive.  Also,  an  armored  or  mechanized  infantry  battalion  can  expect  to 
train  there  only  once  every  18  months  or  so,  a  timetable  similar  to  that  of  an  Air  Force  or 
Navy  squadron  training  at  Red  Flag  or  Top  Gun.  The  benefits  of  the  training  are  judged  by 
senior  military  leaders  to  justify  the  costs;  their  major  complaint  is  that  there  are  not 
sufficient  resources  (land  and  air  space,  as  well  as  budgetary  authority)  to  support  several 
Top  Guns,  Red  Flags,  and  National  Training  Centers  so  that  troops  could  train  there  more 
frequently. 

2.  Collective  Training — With  Simulator  Networking 

Starting  in  the  early  1980s,  an  R&D  effort  to  harness  the  technology  needed  for  a 
simulation  network  that  would  provide  effective  and  affordable  resource  lor  collective 
training  was  jointly  supported  by  the  Army  and  the  Defense  Advanced  Research  Projects 
Agency  (DARPA).’ 

Simulator  Networking  (SIMNET)  was  initiated  as  a  DARPA  project  on  large-scale 
simulator  networking  in  1983.  It  is  a  proof-of-principle  technology  demonstration  of 
interactive  networking  to  combine  many  individuals  or  crews  for  man-in-the-loop,  real¬ 
time,  battle-engagement  simulation  and  wargaming.  It  is  the  first  system  to  achieve  true 
interactive  simulator  networicing  for  the  collective  training  of  combat  skills  in  military  units 
from  mechanized  platoons  to  battalions.  SIMNET  is  also  adaptable  for  training  or 
exercising  commanders  and  staffs  at  higher  echelons,  usable  in  the  development  of  military 
concepts  and  doctrine,  and  suitable  for  the  testing  and  evaluation  of  altemau.e  weapon- 
system  concepts  prior  to  acquisition  decisions. 

In  order  to  undertake  frequent  training  for  large  engagements,  SIMNET  and 
SIMNET  follow-ons  have  been  consciously  designed  to  trade  some  “objective  reality”  in 
order  to  maintain  selective  fidelity  for  the  features  needed  to  provide  effective  training.  As 


For  a  mote  complete  discussion,  see  Reference  [18]  or  [34). 
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mentioned  earlier,  a  common  subjective  assessment  is  that  SIMNET  achieved  60%  fidelity 
at  10%  of  actual  equipment  costs. 

As  of  1  January  1990,  the  available  SIMNET  components  consisted  of  about  260 
ground  vehicle  and  aircraft  simulators,  communications  networks,  command  posts,  and 
data  processing  facUities  distributed  among  nine  sites — five  in  the  continental  United  States 
(CONUS)  and  four  at  U.S.  Army  locations  in  Europe  (USAREUR).*®  A  first-order 
estimate  of  the  cost  of  this  program  to  this  point  is  $200  million,  $50  million  in  research, 
development,  tCvSt  and  evaluation  (RDT&E)  and  $15  million  in  procurement  funds  [35]. 

SIMNET  technology  has  been  transferred  to  the  Army  as  SIMNET-T,  a  collective 
or  imit  training  capability  that  the  Army  is  planning  to  extend  Army-wide  through  a  large- 
scale  follow-on  acquisition  program.  Specifically,  SIMNET  was  “completed”  and 
transferred  to  the  Army  starting  in  October  1989.  Since  then,  the  Army’s  Project  Manager, 
Training  Devices  (PM  TRADE)  has  been  preparing  for  procurement  of  the  first  several 
hundred  units  of  the  Close  Combat  Tactical  Trainer  (CCTT)  system,  the  production  follow- 
on  to  SIMNET. 

This  CCTT  procurement  is  to  be  the  first  production  buy  of  what  is  envisioned  to 
be  several  thousand  networkable  simulators  for  armor,  m'*chanized  infantry,  aviation,  and 
air  defense  units.  When  completed,  the  planned  training  system  will  provide  a  company- 
level,  SIMNET-like  training  capability  at  each  Army  battalion’s  home  base  to  provide 
individuals  with  20  days  of  CCTT  time  per  year.  In  addition,  1 1  tank  and  10  infantry 
platoon  mobile  simulators  will  be  used  by  reserve  components.  Also,  reserve  components 
will  be  able  to  use  the  Army’s  CCTT  facilities  on  some  weekends. 

SIMNET-D,  another  version  located  at  Fort  Knox,  Kentucky,  and  AIRNET-D,  at 
Fort  Rucker,  Alabama,  provide  a  developmental  capability  that  can  be  reconfigured  to 
simulate  new  design  concepts  for  evaluation  in  SIMNET  trials.  These  form  the  basis  of  a 
new  joint  Army-DARPA  initiative  to  demonstrate  Advanced  Distributed  Simulation 
Technology  (ADST)  for  use  in  system  development,  studies,  and  analyses  [36].  In  fact, 
PM  trade’s  acquisition  plan,  as  reported  in  October  1989,  includes  provisions  for  these 
air  and  ground-vehicle  developmental  test  beds  (AIRNET-D  and  SIMNET-D)  intended  to 


The  CONUS  s'tes  are  at  Fort  Knox,  Kentucky,  Fort  Benning,  Georgia,  Fort  Rucker,  Alabama, 
Cambridge,  Massuchusetts,  and  Washington,  D.C.  The  USAREUR  sites  are  in  West  Germany  at 
Grafenwdhr,  Friedberg,  Schweinfurt  and  Fulda.  The  Fort  Knox  site  is  currently  the  largest  SIMNET 
facility  with  simulators  for  44  Ml  Abrams  tanks.  28  M2/3  Bradley  Fighting  Vehicles,  2  Scout/ Attack 
Helicop^,  2  Close  Air  Support  Fighter  Aircraft,  a  Battalion  Task  Force  Tactical  Operations  Center, 
an  Administrative-Logistics  Operating  Center,  and  other  command  and  control,  ai’Jlleiy  and  mortar-fire, 
and  close  air  support  control  elements— all  fully  interactive  on  a  local  area  network. 
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“provide  materiel  developers  the  ability  to  try  out  their  ideas  prior  to  issuing  doctrinal 
changes  or  the  bending  of  metal”  ([36],  p.  14). 

Efforts  are  currently  being  made  to  network  existing,  higher  fidelity  simulators. 
Examples  of  this  are  the  linking  of  Air  Force  and  Navy  flight  simulators  described  in 
Reference  [37],  and  the  linking  of  different  Army  helicopter  simulators  described  in 
Reference  [38].  As  these  simulator  networking  concepts  mature,  the  capability  to  link  these 
higher  fidelity  simulators  to  SEMNET/AIRNET  will  likely  be  developed. 

3 .  Collective  Training — Of  Higher  Echelon  and  Joint  Commands 

In  1988,  the  Defense  Science  Board  (DSB)  Task  Force  on  Computer  Applications 
to  Training  and  Wargaming  summarized  its  view  on  the  issue  of  the  collective  training  of 
military  higher  echelons  and  Joint  Commands  as  follows  ([39],  p.  1): 

Computer-based,  simulated  scenarios  offer  the  only  practical  and  affordable  means  to 
improve  the  training  of  joint  operational  commanders,  their  staffs,  and  the  commanders 
and  staffs  who  report  to  them.  Such  decision  makers  need  the  opportunity  to  exercise 
their  decision  skills,  to  test  war  plans,  and  to  train  to  work  as  a  closely  coordinated  force. 

Increasingly,  joint  training  cannot  be  conducted  in  the  anticipated  theater  of  operations. 

There  are  political  objections  to  disruption  of  civil  activity.  The  cost  of  an  actual 
exercise  at  this  level  is  great  Battle  simulation  offers  the  only  opportunity  to  practice 
the  use  of  certain  weapon  systems,  sensors,  tactics,  and  techniques  against  a  skilled 
adversary. 

Today,  it  is  possible  to  make  a  substantive  improvement  to  computer-assisted  joint 
training  and  wargaming.  The  addition  of  the  Vice  Chairman  to  the  Joint  Chiefs  of  Staff, 
with  the  delegated  responsibility  to  oversee  CINC  [i.e.,  commander-in-chief]  operational 
planning  and  to  serve  as  spokesman  for  the  CINCs,  provides  a  management 
opportunity.  There  is  also  a  technological  opportunity.  Driven  by  the 
commercial  market,  computer  technology  is  evolving  rapidly  and  is  increasingly  cost- 
effective.  For  the  most  part,  the  computer  technologies  supporting  computer-based 
training  are  adequate.  The  challenge  is  to  take  cost-effective,  but  maximum  advantage  of 
these  technologies. 

The  task  force  considered  simulation,  gaming,  and  training  to  improve  the  readiness 
of  joint  commanders  and  staffs.  They  did  not  address  individual  training,  nor  service- 
specific  collective  training.  However,  they  did  review  the  services’  collective  training 
conventions  and  facilities,  as  well  as  the  available  and  emerging  technological  capabilities 
applicable  to  joint  training,  and  concluded  generally  as  follows  ([39],  p.  2): 

•  Each  service  and  the  JCS  build  their  own  simulations. 

•  Redundancy  and  overlap  of  simulations  are  common. 

•  Separate  simulations  lack  joint  training  functionality.  While  simulations 
typically  models  service-relevant  forces,  threats,  and  operating  environment 
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well,  they  do  not  take  advantage  of  other  service’s  efforts,  and  hence  are  often 
inconsistent. 

•  The  simulations  do  not  take  advantage  of  computing  and  communications 
technology  to  have  distributed  training  that  avoids  wasting  .resources  and  time 
through  unnecessary  travel. 

The  task  force  then  made  five  complementary  recommendations:  (a)  make  joint 
simulations  interoperable,  (b)  promote  joint  simulation  usage,  (c)  establish  requirements  for 
future  capabilities,  (d)  establish  a  prototype  program,  and  (e)  undertake  a  major  joint 
training  initiative.  Appendix  H,  “Actions  Required  for  Implementation,”  in  the  task  force’s 
report  detailed  the  specific  actions  to  be  taken  to  effect  the  recommended  steps. 

There  have  been  isolated  cases  where  parts  of  tliese  recommendations  have  been  put 
into  effect.  For  example,  relative  to  previous  exercises,  the  1990  Return  of  Forces  to 
Germany  (REFORGER)  exercises  emphasized  computer  simulation  to  train  higher 
echelons,  and  de-emphasized  the  use  of  larger  numbers  of  personnel  moving  realistically 
(i.e.,  destructively)  over  the  countryside.  This  exercise  is  analyzed  in  [40].  Unfortunately, 
the  analysis  emphasizes  narrowly  defined,  easy-to-measure  cost  saxtings  (which  are  small), 
rather  than  broadly  defined,  difficult-to-measure  benefits  of  th’s  sort  of  training  on  the 
performance  of  higher  echelon  leaders  and  managers. 
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IV.  DATA  AND  ANALYSES 


This  section  reviews  OPTEMPO  and  simulator  use  in  the  military  services. 
Previous  research  on  individual  and  collective  training  simulation  is  summarized,  and  new 
information  is  presented.  These  data  are  then  used  to  perform  cost-effectiveness  analyses 
of  the  use  of  simulators  in  individual  and  collective  training. 

A.  DATA 

The  DoD’s  formal  system  of  military  education  and  training  is  focused  nearly 
exclusively  on  individuals  within  military  schools  and  training  institutions  or  centers.  An 
annual  report  of  such  training,  the  Military  Manpower  Training  Report  (MMTR)  [1],  is 
submitted  to  the  Congress  by  the  Secretary  of  Defense  (SECDEF)  in  accordance  with  10 
U.S.C.  138(d)(2).  For  1989,  the  last  year  of  actual  data  in  the  report,  approximately  6.7% 
of  the  DoD  budget,  and  14,2%  of  active  military  personnel,  are  receiving,  giving,  or 
supporting  individual  training.  However,  some  individual  on-the-job  training  (OJT)  and 
essentially  all  collective  military-operations  training  occurs  within  the  services’  operational 
mission  units.  There  is  no  comprehensive  DoD  report  for  collective  training  similar  to  that 
for  individual  training. 

One  rough  estimate  of  this  training  expenditure  is  operating  tempo  (OPTEMPO). 
OPTEMPO  costs  are  justified  largely  in  terms  of  the  collective  training  that  is  necessary  to 
gain  and  maintain  satisfactory  levels  of  force  readiness — the  ability  of  a  military  force  or 
uiiit  to  fight  and  win  in  combat.  The  estimated  OPTEMPO  in  the  President’s  Budget  for 
FY  1991  (prior  to  the  Desert  Shield  and  Desert  Storm  operations)  is  shown  in  Table  2. 

Detailed  estimates  of  the  associated  costs  are  difficult,  but  rough  estimates  of  the 
variable  costs  of  OPTEMPO  are  possible  from  budget  documents.  For  example,  Table  3 
shows  estimates  for  flight  OPTEMPO  for  the  three  military  departments.  Table  4,  for  ship 
operations  and  maintenance,  and  Table  5,  for  Army  vehicle  OPTEMPO.  The  total  of  $21.4 


Active  military  students,  183.8  thousand;  direct  military  support.  89.2  thousand;  BOS  and  HQ,  30.1 
litousand;  Total  Individual  training  budget,  $19.4  billion  [1].  Total  DoD  budget,  $290.8  billion  [41]. 
Total  active  force,  2,131  thousand  [42]. 
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billion  from  Tables  3  through  5  is  useful  in  that  it  shows  costs  that  would  decrease  due  to 
reductions  in  OPTEMPO. 


Table  2.  OPTEMPO  for  Various  Services  and  Systems,  FY  1991 


Unit  and  Catettorv 

OPTEMPO 

Flying  Hours  Per  Crew  Per  Monti, 

Army  Tactical  Helicopter 

14.5 

Navy/Marine  Tactical  Air/Antisubmarine  Warfare 

25.0 

Air  Force  Tactical  Air 

19.5 

Air  Force  Strategic  Air 

17.6 

Air  Fbrce  Airiift 

30.2 

Navy  Steaming  Days  Per  Quarter 

Dqtloycd  Fleets 

50.5 

Nan-Deployed  Fleets 

29.0 

Army  Combat  Vehicle  Mileage  Per  Year 

800.0 

teurce:  Reference  [43]. 

Table  3.  OPTEMPO  Costs  and  Flying  Hours  for  the  Services,  FY  1991 

Cost 

(Billions  of  FY  Flying  Hours 


Category 

1991  Dollars! 

(Millions' 

Air  Fbrce  Total 

$6.5 

3.4 

Peacetime  Operating  Stocks 

2.3 

Industrially  Funded  Maintenance 

2.9 

Aircraft  Petroleum,  Oil,  and  Lubricants 

1.3 

Navy  Total 

$4.1 

2.2 

Aircraft  Operations 

2.3 

Aircraft  Maintenance  and  Engineering 

1.8 

Army  Total 

$1.1 

1.7 

Aircraft  Petroleum,  Oil,  and  Lubricants 

0.1 

Consumables 

0.2 

Rqnir  and  Depot  Maintenance 

0.8 

DoD  Total 

$11.7 

7.3 

Source*;  Co»ts:  Air  Force,  Reference  (44],  p.  A-32;  Navy.  Reference  (45];  and  Army, 
Reference  (46),  with  an  adjustment  to  Repairables  and  Depot  Spues.  Flying  hours;  Air 
Force.  Reference  (44),  FY  1989  data;  Navy,  Reference  (8);  and  Army,  Reference  (47). 


Table  4.  Ship  Operations  and  Maintenance,  FY  1991 


Cost 

(Billions  of  FY 
1991  Dollars 


Ship  Operations 
Ship  Maintenance 

Ship  and  Weapon  System  Maintenance  and  Engineering 
Total 


Source:  Reference  (4S]. 
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Table  5.  Army  Vehicle 

OPTEMPO.  FY  1991 

Cost 

(Billions  of  FY 

Catecorv 

1991  Dollars 

Fuel 

Consumables 
Repairable  Spares 

Depot  Maintenance  of  Combat  Vehicles 
Total 


Source;  Reference  [46],  p.  36-43. 


.)espite  the  scope  of  such  training  implicit  in  this  OPTEMPO,  there  has  been 
relatively  little  technology-base  research  and  development  (R&D)  devoted  to  it.  Most  of  the 
training-technology  R&D  in  the  Training  and  Personnel  Systems  Technology  program  has 
traditionally  been  aimed  at  the  other  military  training  domain — that  of  individi  al  training  in 
institutions.  This  has  been  recognized  for  at  least  15  years,  as  evidenced  by  the  attention 
given,  and  recommendation  made  for  increasing  technology- base  R&D  in  support  of 
collective  training,  by  no  fewer  than  four  DSB  studies  [39, 48, 49,  and  50]. 

B.  COST-EFFECTIVENESS  COMPARISONS 

This  section  reports  results  from  a  literature  review  of  simulator  costs  and 
effectiveness.  The  literature  reports  results  for  representative  flight,  maintenance,  and 
collective  training  simulators.  Two  preliminary  cost-effectiveness  comparisons  are 
generated.  The  first  cost-effectiveness  comparison  shows  the  trade  flying-hour  OPTEMPO 
for  possible  increases  in  simulator  time.  The  second  compares  the  costs  and  effectiveness 
of  SIMNET  and  field  exercises.'^  In  addition,  we  briefly  summarize  an  interesting 
analysis  of  the  effects  of  home-station  exercises  on  NTC  effectiveness. 

The  purpose  of  this  review  is  to  highlight  key  information  that  is  needed  to  develop 
cost-effectiveness  tradeoffs  rather  than  to  present  definitive  answers.  However,  based  on 
this  preliminary  analysis,  the  key  conclusions  of  this  comparison  are  that  each  approach 
(flight  simulators  versus  flying  hours,  SIMNET  versus  home-station  field  exercises)  has 
tasks  for  which  it  is  much  more  effective  than  the  others  as  a  training  approach,  that  there 
are  tasks  for  which  both  can  be  used,  and  that  simulators  and  simulations  offer  an 


We  hoped  to  compare  costs  and  effectiveness  of  NTC  and  a  SIMNET-like  facility.  It  was  not  possible 
to  obtain  comparable  effectiveness  data.  A  summary  of  the  cost  data  collected  is  i.icluded  in 
Appendix  A. 
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opportunity  to  cost-eff"clively  augment  actual  equipment  or  field  exercise  Gaining.  Greater 
emphasis  on  simulators,  and  less  on  using  actual  equipment  appears  to  be  warranted. 

1 .  Previous  Research  on  Individual  and  Collective  Training — With 
Simulators 

Simulators  for  individual  operators  and  maintainers  have  a  long  history  in  military 
training,  and  have  been  widely  studied.  One  summary  of  these  studies  [16]  is  itself  a 
summary  of  both  a  decade  of  research  at  IDA  as  well  as  the  work  of  many  other  authors.  A 
later  study  surveys  more  recent  analyses  [17],  A  few  of  the  cost,  effectiveness,  and  cost- 
effectiveness  results  cited  there  are  presented  below. 

a.  Simulators  in  the  Training  Taxonomy 

Simulators  are  widely  used  in  training.  Figure  4  shows  examples  of  where  various 
types  of  simulators  fit  within  a  slightly  expanded  version  of  the  training  taxonomy 
intrr  luced  in  Figure  1.  Embedded  training  uses  actual  equipment  being  stimulated  for  the 
purposes  of  training.  Actual  equipment  and  exercises  are  increasingly  used  for  training  as 
one  moves  from  the  upper  left  to  the  lower  right  of  the  matrix.  Examples  of  these 
simulations  are  discussed  below. 


School  Continuation  Unit 


Maintenance 

Flight 

Embedded 

training, 

portable 

part-task 

trainers 

Wargaming, 
crew  training 
(e.  g,.  C-130) 

Embedded 

training, 

networked 

simulators 

Figura  4.  Simulators  Placed  In  the  Training  Taxonomy 
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b.  Flight  Simulators 

The  investment  cost  for  individual  simulators  is  often  similar  to  the  cost  of  actual 
equipment.  However,  Orlansky,  Knapp,  and  String  [51]  report  that  the  operating  and 
support  (O&S)  costs  for  39  simulators  compared  with  actual  aircraft  using  1980-81  data 
was  an  order  of  magnitude  lower.  That  is,  O&S  costs  for  the  median  simulator  was  8%  of 
O&S  costs  for  actual  equipment,  and  the  unweighted  mean  cost  ratio  was  11%.  The  ratios 
are  skewed  to  the  right  in  that  the  range  was  from  2%  to  59%,  the  standard  deviation  was 
0.1 1,  and  34  of  the  39  observations  are  within  one  standard  deviation  of  the  mean. 

The  life-cycle  cost  comparisons  were  not  reported  in  most  of  these  studies. 
However,  fragmentary  evidence  supported  a  time  to  breakeven  of  one  to  two  years.  If  we 
use  an  O&S  cost  ratio  of  10%  and  equal  procurement  prices,  discounted  life-cycle  costs  of 
simulators  arc  as  large  as  25%  of  actual  equipment  costs 

Cost  ratios  for  different  mixes  of  simulator  and  actual  equipment  training  depend  on 
whether  there  is  excess  capacity  available.  If  both  simulators  and  actual  equipment  usage 
can  be  increased,  the  cost  ratio  would  be  the  O&S  cost  ratio.  10%.  If  the  choice  is  between 
purchasing  one  or  the  other,  the  cost  ratio  is  25%.  If  actual  equipment  is  already  available 
but  simulators  must  be  purchased,  the  cost  ratio  is  33%. 

Simulator  effectiveness  depends  on  the  task  being  trained.  Orlansky  and  String 
[52]  report  the  results  of  a  study  that  compiled  34  Transfer  Effectiveness  Ratios  (TERs)  in 
22  studies  showed  a  median  value  TER  of  0.41,  a  mean  TER  of  0.52,  and  a  standard 
deviation  of  0.45.  Comparing  these  TERs  with  the  cost  ratios  discussed  above,  59%  of  the 
TERs  were  greater  than  the  0.33  cost  ratio,  TO'^  of  the  TER.«  were  greater  than  or  equal  to 
the  0.25  cost  ratio,  and  94%  were  greater  than  or  equal  to  the  0.1  cost  ratio  derived  above. 
The  size  of  the  TERs  was  also  correlated  to  characteristics  of  the  task  to  be  trained. 
Therefore,  inferences  can  be  drawn  about  what  tasks  should  be  trained  mainly  on 
simulators,  and  what  tasks  should  be  trained  mainly  on  actual  equipment 

There  is  evidence  that  the  marginal  contribution  of  simulator  time  to  the 
performance  of  private  pilots  declines  as  more  of  it  is  used  [15].  The  results  of  that  repon 
are  presented  in  Table  6.  TTie  cumulative  hours  saved  are  always  based  on  comparison 
with  the  control  group  (no  simulator  time).  For  example,  7  simulator  hours  save  6.2  flight 
hours,  so  the  cumulative  TER  is  0.9.  The  incremental  hours  saved  in  any  given  row  are 
based  on  a  comparison  with  the  row  above  it.  For  example,  the  incremental  savings  of  1 .6 
hour  in  the  7  simulator  hours  row  indicates  that  the  additional  4  hours  of  simulator  time 
resulted  in  1.6  hour  less  flying  time  needed  to  reach  a  given  level  of  proficiency.  This 
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means  that,  over  that  interval,  each  simulator  hour  saved  0.4  flight  hour;  hence,  that  is  the 
incremental  TER.  The  cost  ratio  between  the  simulator  and  the  airplane  is  0.57,  with  the 
ratio  rising  to  0.73  if  instructor  time  for  both  training  approaches  is  included.***  Therefore, 
in  this  example  the  optimal  amount  of  simulator  time  is  4-5  hours. 


Table  6.  Transfer  Effectiveness  of  a  Ground-Based  General  Aviation  Trainer 


Total  Hours _ Hours  Saved _ Transfer  Ratios 


Simulator 

Flifihl 

Cumulative 

Incremental 

Cumulative 

Incremental 

0 

3 

44.5 

39.9 

4.6 

4.6 

1.5 

1.5 

7 

38.3 

6.2 

1.6 

0.9 

0.4 

11 

37.3 

7.2 

1.0 

0.7 

0.2* 

Source:  Reference  [IS]. 

Notes:  Flight  hours  are  the  hours  needed  to  reach  a  speciTied  level  of  pilot  performance.  Hours 

and  transfer  ratios  to  reach  final  flight  check  were  essentially  identical. 

*  Apparent  error  is  due  to  rounding. 

Care  should  be  taken  in  applying  these  data  to  the  military  because  this  experiment 
was  done  on  a  low-technology,  civilian  aircraft  and  simulator,  and  small  sample  sizes  in 
this  experiment  result  in  low  statistical  significance.  However,  the  pattern  of  the  results  is 
consistent  with  both  intuition  and  the  standard  concepts  of  diminishing  marginal  returns. 

The  material  in  [IS]  also  suggests  that  care  be  taken  in  using  the  TERs  reported  in 
[16].  They  should  be  applied  with  caution  if  one  is  considering  dramatic  changes  in  the 
division  of  time  spent  on  simulators  and  actual  equipment  However,  in  an  era  of  budgetary 
stringency,  marginal  increases  in  simulator  time  appear  to  be  cost-effective. 

c.  Maintenance  Simulators 

The  results  reported  in  [16]  for  these  simulators  are  not  as  numerous  or  reliable  as 
for  flight  simulators.  However,  u-Sing  the  criterion  of  test  performance  during  training, 
simulators  were  at  least  as  good  as  actual  equipment  (TERs  of  1.0),  and  the  investment  cost 
ratios  were  roughly  in  the  range  of  0.5.  Using  data  from  other  sources  [53],  O&S  costs 
for  maintaining  simulators  are  usually  much  less  than  actual  equipment 

There  are  two  aspects  to  the  studies  summarized  here  that  also  tend  to  bias  the 
results  against  simulators.  First,  with  respect  to  cost  avoidance,  simulators  often  allowed 
students  to  learn  faster,  resulting  in  less  time  in  training.  The  cost  savings  in  pay  and 
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It  is  not  clear  from  the  text  whether  these  ratios  are  life  cycle  or  only  operating  costs. 


allowances  were  not  usually  included  in  the  cost  ratios  reported  in  [16]  or  [17].'^  Second, 
with  respect  to  the  tasks  trained,  TER  comparisons  require  that  training  activities  be 
performed  on  both  actual  ejquioment  and  simulators.  One  of  the  strengths  of  simulators  is 
that  they  can  train  tasks  that  are  not  trained  on  actual  equipment  due  to  danger  or  cost. 
These  training  advantages  are  not  reflected  in  the  reported  TERs. 

d.  Collective  Trainiug  Siirruiators 

The  Navy  has  for  some  years  employed  computer-based  simulations  in  its  fleet 
operational  training  facilities  in  San  Diego,  California,  and  at  Dam  Neck  in  Virginia  Beach, 
Virginia.  These  simulations  typically  involve  ships  afloat  as  well  as  the  simulations. 

The  Air  Force’s  Tactical  Air  Command  has  used  contract  simulation  facilities  in  the 
aerospace  industry  for  the  collective  training  of  three-element  teams — two  aircraft  pilots  and 
a  ground-based  radar  intercept  controller  or  an  airborne  warning  and  control  system 
(AWACS)-based  controller.  The  training  covers  a  wide  range  of  tasks,  including  electronic 
combat  and  all-aspect  missile  defense.  For  safety,  range  and  environment,  coordination, 
and  security  reasons  this  training  is  not  done  on  actual  equipment. 

Two  different  studies  of  collective  training  were  undertaken  using  these  facilities 
[54  and  55].  In  the  first  study,  the  pilots  were  from  Air  National  Guard  and  fighter 
interceptor  squadrons.  In  the  second  study,  the  pilots  were  from  active-duty  tactical  fighter 
wings.  Pre-  and  post-training  surveys  were  used  to  determine  what  tasks  needed  to  be 
gained,  and  how  the  value  of  existing  unit  training  compared  with  the  simulator  training. 

i  Comparisons  of  perceived  training  effectiveness  for  the  pilots  and  controllers  are 
reported  in  Table  7.  For  example,  in  the  1990  study  for  pilots  [54],  of  the  41  identified 
tasks,  21  were  better  trained  by  the  simulator,  8  were  equally  well  uained  by  the  simulator, 
6  were  better  trained  in  unit  training,  and  6  were  not  measured  in  the  simulator.  In 
summary,  71%  of  all  enumerated  tasks  were  trained  at  least  as  well  in  the  collective  training 
simulator. 

Pilots’  opinions  on  why  these  results  occurred  centered  around:  a)  repeated  practice 
within  a  brief  time  to  try  alternative  tactics  and  reinforce  positive  habits,  b)  immediate 
performance  feedback  in  the  form  of  real-time  kill  removal  ([54],  p.  19).  For  controllers, 
the  key  element  was  the  ability  for  immediate  after-action  reviews  with  the  pilots  they  were 
supporting  to  understand  “the  content  and  timing  of  information  required  by  pilots  during 


For  a  further  discussion  of  this  point,  see  Reference  [3]. 
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air  combat  engagements.”  ([54],  p.  23)  The  subjective  opinions  expressed  in  [55]  were 
similar  to  those  from  [54],  and  a  correlation  of  these  opinions  with  pilot  characteristics, 
such  as  current  unit  type,  qualification,  and  F-15  flying  hours  showed  no  significant 
relationships. 


Table  7.  Effectiveness  Results  for  F-15  Multiship/Controller  Training 


Respondents/ 

Source 

Simulator 

Better 

No 

SigniHcant 

[Mfference 

Unit 

Training 

Better 

Not 

Measured  in 
Simulator 

Totals 

Percentage 
Simulator 
as  Good 
or  Better 

Pilots 

1541.  p.  15 

21 

8 

6 

6 

41 

71 

155],  p.  22« 

11 

15 

4 

30 

87 

Controllers 

[54],  p.  63-64 

34 

6 

40 

85 

[55],  p.  71 

15 

5 

20 

100 

*  Reference  [SS]  presents  these  data  in  a  way  that  does  not  allow  this  aggregation  to  be  made  with 
certainty.  Therefore,  to  aggregate  in  this  way,  all  uncertainties  were  decided  favoring  continuation 
training  over  simulator  training. 


If  it  were  done  on  actual  equipment,  the  cost  per  flying  hour  for  two  F-15s  is  in  the 
range  of  $10,(X)0,  while  the  simulation  facility  is  available  for  approximately  $1,000  per 
hour  [56], 

2.  Flying  Hours  and  Flight  Simulators 

Examining  DoD-wide  tradeoffs  of  flying  hours  and  flight  simulator  use  is 
inherently  difficult  The  services  have  peacetime  missions  that  require  flying,  and  this 
flying  does  contribute  to  traimng.  In  addition,  various  aircraft  categories  and  organizations 
are  flying  at  different  proportions  of  their  training  requirements,  and  different  aircraft 
categories  are  already  being  trained  with  different  mixtures  of  actual  and  simulated 
equipment  However,  budgetary  realities  encourage  a  better  understanding  of  the  effects  of 
actual  and  simulated  flight  on  the  training  of  personnel. 

A  5%  decrease  in  aircraft  OPTEMPO  (approximately  365,{)(X)  flying  hours)  would 
save  $585  million.  If  this  money  were  applied  to  operating  flight  simulators,  it  would 
increase  the  number  of  simulator  hours  by  three  to  ten  times  the  amount  of  flight  hours 
given  up.  TERs  seem  to  support  the  contention  that  this  reallocation  would  be  cost- 
effective,  but  the  phenomena  of:  (a)  differential  simulator  use  across  services  and 
platforms,  and  (b)  declining  incremental  training  effectiveness  suggests  that  larger  changes 
in  OPTEMPO  and  simulator  use  be  implemented  only  if  the  early  results  of  focused 


experimentation  and  analysis  warrant.  Any  decrease  in  aviation  OPTEMPO  would  have  to 
be  tailored  to  the  specific  types  of  aircraft  involved. 

a.  Current  Simulator  Usage  and  Costs 

Our  best  estimates  of  simulator  usage  are  as  follows.  For  Anny  simulator  use, 
operating  cost  data  and  manipulation  of  simulator  hour  information  provided  in  [38]  are 
shown  in  Table  8.  The  simulator  hours  per  month  is  a  rough  analog  to  flying  hours  per 
month  in  the  OPTEMPO  information  cited  previously.  Imprecision  occurs  because  some 
simulators  have  two  station  cockpits  that  can  be  used  to  train  either  one  crew  or  two 
individuals  at  the  same  time.  Also,  these  data  cover  a  combination  of  simulators  at  Fort 
Rucker,  where  simulators  are  often  scheduled  for  two  shifts  a  day,  and  simulators  at  other 
locations,  where  such  tight  scheduling  is  more  difiicult.  Also  shown  are  FY  1989 
operating  costs  for  both  simulators  and  actual  equipment. 


Table  8.  Army  Helicopter  Simulator  Hours  and  “Per  Cockpit”  Operating  Costs 
for  Simulators  and  Actual  Equipment,  FY  1989 


Flight  Simulator 


Category 

Total  Hours 
(Thousands) 

Hours 
per  Month 
per  Seat 

Cost 
per  Hour 

Actual 
Equipment 
Cost  per  Hour 

Simulator 

Cost 

Percentage 

UH-1  Instrument  Flight  Trainer 

168.8 

185 

$21 

$518 

4% 

UH-^  Operatianal  Flight  Trainer 

25.4 

132 

$123 

$1,058 

12% 

CH-47  Operational  Flight  Trainer 

6.7 

121 

$229 

$1,688 

14% 

AH-1  Operational  Flight  Trainer/ 

26.1 

161 

$241 

$1,108 

22% 

Weapons  System  Simulator 

AH-64  Combat  Mission 

27.0 

225 

$125 

$3,055 

4% 

SimulattM' 

Averages 

- 

164 

$148 

$1,485 

11% 

Source:  Reference  [57]. 

Note;  Total  timulator  hours  is  254,000. 


Firm  procurement  cost  data  for  these  simulators  is  difficult  to  obtain  because 
estimates  depend  on  the  number  procured  and  the  level  of  engineering  changes  allowed  for 
any  given  procurement.  However,  discussion  with  individuals  in  the  PM  TRADE 
organization  [58]  support  the  contention  that  a  full  motion  simulator  with  good  visual 
fidelity  will  cost  jqjproximately  as  much  as  one  copy  of  the  actual  equipment 

A  rough  cost  analysis  can  be  done  using  these  data.  An  AH-64  combat  mission 
simulator  costs  in  the  range  of  $15  million  per  copy.  The  O&S  costs  per  cockpit-year  are 
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$0,675  million  (5,400  hours  times  $125  per  hour).  The  undiscounted  lifetime  cost  of  the 
simulator  is  $25.1  million,  and  the  discounted  present  value  is  $20.1  million. 

This  purchase  would  be  paid  for  by  reducing  flying  hours.  The  reduction  would 
depend  on  the  desired  budgetary  impact.  A  pay-as-you-go  policy  would  involve  giving  up 
4,910  hours  spent  in  year  1  (4,910  hours  at  $3,055  per  hour  yields  the  $15  million 
purchase  price)  and  221  flying  hours  in  years  2  through  16.  An  alternative  is  to  invest  in 
year  1  and  pay  back  865  flying  hours  per  year  in  years  2  through  16. 

The  cost-effectiveness  evaluation  would  hinge  on  whether  81,000  additional 
simulator  hours  were  as  effective  in  training  AH-64  pilots  as  the  8,225  flying  hours  traded 
for  the  simulation  time  in  the  pay-as-you-go  plan  or  the  12,975  flying  hours  traded  in  the 
invest  now,  pay  later  plan. 

A  comparison  of  simulator  hours  to  total  flying  hours  for  five  different  Army 
helicopters  is  shown  in  Table  9.  Helicopter  type,  simulator  hours,  flying  hours,  and  the 
proportion  of  cockpit  hours  spent  in  simulators  arc  shown. 


Table  9.  Comparison  of  Flying  and  Simulator  Hours 
for  Selected  Army  Htllcopters,  FY  1989 


Helicooter 

Simulator  Hours 
(Thousands) 

Flying  Hours 
(Thousands) 

Percentage  of  Cockpit 
Hours  in  Simulators 

AH-1 

26.1 

140.7 

15.6 

AH-64 

27.0 

69.7 

27.9 

CH.47 

6.7 

52.4 

11.3 

UH-1 

168.8 

692.1 

19.6 

UH-60 

25.5 

191.9 

11.7 

Sources;  SimuUun-  hows.  Reference  (57];  flying  hours,  Reference  [47]  and  supporting  dau. 


While  detailed  data  have  not  yet  been  published,  we  have  some  subjective 
evaluations  of  the  proporuon  of  simulator  hours  to  total  hours  that  could  be  effectively  used 
in  Army  helicopter  training  [59].  These  are  24-25%  for  the  UH-60, 28-29%  for  the  CH- 
47,  and  31-32%  for  the  AH-1.  The  fact  that  the  percentages  of  cockpit  hours  for  these 
three  helicopters  shown  in  Table  9  are  lower  than  the  percentages  reported  by  in  [58] 
suggests  areas  where  simulator  time  could  be  substituted  for  flying  hours. 

For  Navy  usage,  according  to  Reference  [45],  2%  of  the  Primary  Mission 
Readiness  activity  is  contributed  by  simulators.  No  simulator  contribution  is  mentioned  for 
Fleet  Readiness  Squadrons  or  Fleet  Support.  We  have  obtained  usage  information  for 
Navy  simulators,  but  have  not  yet  analyzed  that  data.  We  know  where  to  obtain  some  Air 
Force  simulator  data,  but  were  not  able  to  do  so  during  the  course  of  the  study. 


b.  Increased  Simulator  Usage — Costs  and  Cost-Effectiveness 

(1)  Costs.  Constructing  equal  cost  combinations  of  simulators  and  actual 
equipment  depends  on  the  mix  of  simulators  that  had  to  be  purchased  versus  those  that 
would  be  more  intensively  used.  Based  on  representative  costs  reported  by  Orlansky  in 
[16]  every  flying  hour  given  up  releases  funds  that  could  be  used  to  increase  possible 
simulator  time  between  3  and  10  hours  over  the  life  cycle.  The  3-hour  estimate  includes 
simulator  acquisition  and  operation  versus  actual  equipment  operation,  the  latter  estimate 
includes  operation  only.** 

Another  way  to  examine  this  reallocation  of  funds  is  through  a  breakeven  analysis. 
Orlansky  reports  [16]  breakeven  points  from  eight  months  to  two  years,  which  is 
consistent  with  the  information  on  the  AH-64  combat  mission  simulator.  Therefore,  if  a 
given  amount  of  funds  were  reallocated  from  flying  hours  to  simulators  for  a  short  period 
of  time,  later  budgets  would  be  able  to  reduce  training  expenditure.  The  effectiveness 
implications  are  discussed  below. 

(2)  Cost-Effectiveness.  With  respect  to  training  effectiveness,  the  TERs 
reported  in  [52]  show  59%  at  or  above  one-third,  and  94%  at  or  above  one-tenth.  This 
indicates  that,  even  under  the  worst  case  scenario,  59%  of  these  tasks  could  be  cost- 
effectively  trained  by  increasing  simulator  usage,  and  depending  on  simulator  availability, 
up  to  94%  of  the  tasks  could  be  cost-effectively  trained  by  increased  simulator  usage.  The 
prevalence  of  TERs  that  are  higher  than  the  ratio  of  simulator  cost  to  OPTEMPO  cost 
means  that  equivalent  training  could  be  achieved  with  less  money. 

c.  Large  Changes  in  Simulation 

The  suggestion  to  decreascj  OPTEMPO  by  5%  and  use  some  of  those  funds  for 
increased  simulator  acquisition  and  operation  would  result  in  a  large  change  in  the  relative 
usage  of  simulators  and  actual  equipment.  While  the  5%  decline  in  OPTEMPO  is  small, 
the  percentage  increase  in  simulator  usage  would  range  between  15%  and  50%,  depending 
on  how  many  simulators  had  to  be  Acquired  and  how  many  were  already  available.  The 
AH-64  example  given  previously  is  eVen  more  favorable. 

Both  production  theory  and  Empirical  work  [15]  suggest  that  changes  of  this 
magnitude  could  alter  the  incremental)  training  transfer  effectiveness  ratio.  Such  a  change 
does  not  refute  the  contention  that  more  simulator  use  appears  to  be  cost-effective  at  this 


The  AH-64  example  shows  a  range  of  6  to  12  even  with  simulator  acquisition  included. 
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time  for  the  current  mixture  of  flying  and  simulator  use.  It  does  counsel  against  the 
assertion  that  large  changes  in  OPTEMPO  can  be  easily  offset  by  still  larger  changes  in 
simulator  use. 

3.  Comparing  a  SIMNET*Like  Facility  with  Exercises  at  Home-Station 
Range  and  Maneuver  Areas 

This  section  lays  out  an  approach  to  performing  a  cost-effectiveness  analysis 
comparing  training  in  a  SIMNET-like  facility  and  home-station  range  and  maneuver  areas. 
The  data  to  fully  accomplish  this  comparison  are  not  currently  available.  However,  useful 
inferences  can  be  drawn  from  the  data  that  are  available.  Particularly  for  the  cost  data 
presented  below,  we  believe  a  fortiori  arguments  can  be  made.  That  is,  the  cost  analysis 
that  follows  shows  that  SIMNET  is  a  cost-effective  alternative  for  a  subset  of  tasks 
currently  trained  by  field  exercises.  We  believe  that  the  cost  data  presented  are  biased  to 
underestimate  the  costs  of  field  exercises,  and  overestimate  the  costs  of  SIMNET. 
Therefore,  SIMNET  is  actually  even  more  cost-effective.  In  addition,  the  missing 
information  suggests  areas  where  further  research  or  recasting  of  existing  data  may  h? 
valuable.*’ 

The  results  reported  here  arc  not  inconsistent  with  the  Army’s  Combined  Arms 
Training  Strategy  (CATS)  [21].  A  major  goal  of  CATS  is  to  provide  guidance  on  how 
training  resources  can  and  should  be  used  to  produce  readiness.  The  briefing  materials  that 
outline  this  su^tegy  explicitly  include  training  aids,  devices,  simulators,  and  simulations  as 
a  key  part  of  the  overall  strategy.  They  also  show  examples  (e.g.,  tank  crews)  where  many 
of  the  “component  parts”  of  training  arc  mainly  provided  through  training  devices  and 
simulations,  while  overall  system  competence  and  battle  competence,  that  is,  the  integration 
of  all  aspects  of  training,  is  provided  through  full  service  exercises. 

a.  Costs  of  Various  Options 

Costs  for  home-station  field  exercises  arc  measured  in  operating  or  incremental  cost 
terms,  and  are  basically  OPTEMPO  costs.  The  cost  for  a  SIMNET-like  facility  will  include 
acquisition  costs.  This  is  because  home-station  facilities  already  exist,  but  the  SIMNET- 
like  facilities  have  not  yet  been  acquired.  Home-station  field  exercises  are  estimated  from 
OPTEMPO  and  ammunition  expenditure  only,  because  detailed  estimates  for  costs  such  as 
transportation  were  not  easily  available  to  us. 


Some  NTC  cost  data  are  provided  in  Ai^^endix  A. 


36 


(1)  Cost  of  a  SIMNET-Like  Facility.  The  life-cycle  costs  of  a  SIMNET-like 
facility  are  shown  in  Table  10.  The  explanation  for  these  estimates  is  provided  in  Appendix 
A,  and  a  more  detailed  time-phasing  of  these  costs  is  shown  in  Tables  A-1  to  A-3.  These 
are  roughly  correct  estimates,  not  meant  to  be  alternatives  to  more  detailed  service 
estimates,  but  useful  in  papers  available  for  unlimited  distribution. 


Table  10.  SIMNET-LIke  Facility  Cost  Estimates 


Facility 

Constant  1991 

Current  Dollars 

Constant  1991, 
Discounted 

CCTT  Fixed  Site 

S900 

$1,435.9 

$355.2 

CCTT  Mobile 

400 

656.5 

146.1 

AVCATT 

2QQ 

1.483.4 

329.2 

Total 

$2,200 

$3,575.8 

$830.5 

Notes:  For  more  deteil,  see  Appendix  A.  CCTT  mean  Close  Combat  Tactical  Trainer. 
AVCATT  means  Aviation  Combing  Arms  Tactical  Trainw. 


(2)  Cost  of  Field  Exercises  that  Could  be  Traded  for  a  SIMNET-Like 
Facility.  It  is  extremely  difficult  to  generate  field  exercise  costs  that  are  meaningfully 
comparable  with  SIMNET  costs.  This  is  because  home-station  training  facilities  vary 
widely  in  capability  and  exercises  vary  widely  in  content.  However,  rough  comparisons 
are  reported  here. 

This  cost  comparison  is  between  the  constant,  discounted  total  life-cycle  cost  of  the 
fixed-site  portion  of  a  SIMNET-like  facility  and  the  proportion  of  OPTEMPO  that  would  be 
required  to  “purchase”  that  SIMNET-like  facility,  where  OPTEMPO  is  taken  as  a  rough 
indicator  of  field  exercise  activity.  The  resulting  percentage  of  time  can  then  be  compared 
to  subjective  or  objective  estimates  of  training  effectiveness. 

Tables  A-1  and  A-4  to  A-6  in  Appendix  A  provide  the  information  necessary  to 
make  this  comparison.  They  show  that  between  8%  and  14%  of  the  OPTEMPO  or 
OPTEMPO  and  training  ammunition  activity  would  be  forgone  in  order  to  release  funds 
that  would  have  the  same  net  present  value  as  those  spent  on  this  portion  of  a  SIMNET-like 
facility.  The  field  exercise  mileage  forgone  in  this  notional  tradeoff  (14%  of  OPTEMPO  is 
approximately  1 10  miles  per  vehicle  per  year)  is  almost  exactly  the  same  as  that  which  is 
estimated  to  be  savable  (as  reported  in  [60]). 

b.  Effectiveness  of  Training  Options 

One  oi  the  classic  difficulties  of  a  cost-effectiveness  analysis  is  generating 
quantitative  and  comparable  effectiveness  measures  for  several  different  options.  This 
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problem  exists  in  these  data,  though  there  are  quantitative  pairwise  comparisons  between 
SIMNET  and  home-station  training. 

(1)  *asks  Trained  Better  by  One  Method  Than  Another.  Each  training 
option  is  better  at  certain  aspects  of  training  than  are  the  other  options. 

•  Networked  simulator  strengths  are  the  ability  to  exercise  standing  operating 
procedures,  basic  formations,  general  team  building  [61],  many  command  and 
control  tasks  [21],  and  the  incorporation  of  opposition  forces.  Also, 
networked  simulators  are  particularly  useful  for  timely  after-action  reviews  and 
ease  of  task  repetition. 

•  Actual  equipment  training  is  superior  for  dealing  with  the  integration  of  human 
vision,  particularly  of  ^distant  indistinct  objects,  and  a  sense  of  motion  or 

orientation  into  the  trainmg  exercise  [61]. 

I 

(2)  Effectiveness  of  Exercise  and  SIMNET  Training.  There  is  little 
quantitative  evidence  available  on  die  relative  effectiveness  of  these  approaches  to  training, 
and  that  which  is  available  is  confounded  by  unfortunate  problems  with  experimental 
design,  missing  data  values,  or  small  sample  sizes.  However,  the  data  that  arc  available 
lead  us  to  believe  that,  for  a  significant  subset  of  tasks,  SIMNET  is  more  effective  than 
home-station  field  exercises  where  ejqual  time  is  spent  in  training  on  the  two  approaches. 

The  U.S.  Army  Armor  and  Engineering  Board  executed  an  experiment  where  it 
generated  a  control  and  an  experimental  group  of  tank  platoons,  gave  them  a  pre-test  to 
determine  their  competence,  trained  the  experimental  group  on  SIMNET  and  the  control 
group  through  standard  field  training  for  six  days,  and  then  tested  the  groups  after  the 
training  [62].  The  groups  were  scored  “Go”  or  “No  Go”  on  a  scries  of  tasks  within  the 
given  exercises,  and  the  prc-  and  post-test  Go  percentages  of  these  scores  are  shown  in 
Table  11. 


Table  11.  Performance  of  Tank  Platoons 
Receiving  Standard  Field  or  SIMNET  Training 


Percentage  of  Tasks  “Go" 

Tvdc  of  Trainina 

Pre-Training 

Post-Training 

Standard  Field  Training 

59 

65 

SIMNET  Training 

73 

84 

Note:  This  presentation  is  suggested  in  Reference  [63]. 


The  percentage  improvement  in  Go  scores  of  the  SIMNET-trained  tank  platoons  is 
greater  than  the  field-exercise  trained  platoons  (11%  versus  6%),  and  statistical  tests  on  the 
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differences  between  the  pre-  and  post-training  Go  percentages  (i.e.,  comparing  59%  and 
73%  versus  65%  and  84%)  indicate  that  the  post-training  percentages  are  less  likely  to  be 
the  same.  However,  the  data  as  shown  combine  exercise  tasks  where  only  a  pre-test  or 
only  a  post-test  score  was  available,  and  include  tasks  that  were  judged  not  trainable  by 
SIMNET.  In  addition,  these  comparisons  may  be  flawed  because  the  post-test  differences 
may  be  due  to  the  faa  that  the  two  groups  were  not  equal  before  the  test. 

We  reanalyzed  the  data  on  which  these  summary  statistics  were  based,  and 
generated  Table  12.*’  In  this  analysis  we  took  advantage  of  the  fact  that  there  were  three 
kinds  of  tasks  being  tested;  those  that  were  judged  fully,  partially,  and  not  trainable  by 
SIMNET.  We  also  removed  all  data  that  did  not  have  both  a  pre-  and  post-test  score.  We 
expected  that  both  SIMNET  and  field  exercises  would  at  least  maintain  proficiency.  The 
interesting  question  was  whether  the  SIMNET-trained  units  would  display  positive 
differences  compared  to  the  field  exercise  units. 


Table  12.  Pre-  and  Post-Test  Percentage  “Go” 
by  Training  Type  and  Tralnablllty 


Training  Type 

SIMNET 

Trainabilitv 

Number  of 
Observations 

Pre-Test 

Percentage 

Post-1  est 
Percentage 

SIMNET 

Full 

91 

58 

78* 

Partial 

30 

100 

87 

Naie 

26 

92 

81 

.  Field  Training 

Full 

100 

44 

47 

Partial 

28 

86 

86 

None 

31 

81 

81 

*  This  chsnge  is  significanl  U  the  0.01  level  of  confidence.  See  Appendix  B  for  a 
more  complete  explanation. 


Only  on  the  tasks  that  were  fully  trainable  by  both  SIMNET  and  field  exercises  was 
there  a  statistically  significant  difference  between  pre-  and  post-test  results.  As  shown  in 
Table  12,  the  difference  was  that  the  experimental  (SIMNET)  group  scored  58%  Go  pre¬ 
test  and  78%  Go  post-test.  The  decreases  in  the  other  SIMNET-trained  tasks  are  not 
statistically  significant,  and  the  changes  in  all  three  control  (field  training)  groups  were 
statistically  zero. 


Summary  of  data  in  [62],  tables  2-2-4  and  2-2-5,  p.  2-6;  statistical  tests  reported  im  p.  2-8. 

Additional  statistical  expertise  and  effort  were  contributed  by  Philip  M.  Lurie  of  IDA.  For  a  more 
complete  description  of  the  analysis,  see  Appendix  B. 
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This  table  indicated  that,  as  a  group,  the  field  training  units  maintained  their 
proficiency,  and  SIMNET-trained  group  gained  proficiency  on  the  fully  trainable  items, 
while  their  proficiency  change  on  the  partially  trainable  and  non-trainable  items  was  not 
statistically  significant.  This  analysis  supports  the  contention  that,  for  a  given  subset  of 
tasks  that  are  fully  represented  in  SIMNET,  and  within  a  given  amount  of  time,  SIMNET 
training  is  more  effective  than  field  exercises. 

In  addition,  the  data  indicate  that  a  mix  of  both  types  of  training  has  the  potential  to 
be  the  most  cost-effective  option.  If  tasks  that  are  fully  SIMNET  trainable  are  trained  to 
criteria  on  SIMNET,  there  would  be  more  time  and  resources  available  to  do  field  training 
for  the  other  tasks  that  are  not  trainable  by  SIMNET.  Results  consistent  with  this  sort  of 
specialization  are  also  reported  in  other  analyses  of  SIMNET-like  facilities  [61],  and  in  a 
comparison  cf  pilots  who  had  flight  simulator  training  before  Red  Flag  exercises  [64]. 

One  of  the  main  reasons  that  SIMNET  is  able  to  generate  better  training  results  is 
believed  to  be  related  to  the  much  greater  number  of  simulated  exercises,  miles  driven,  and 
ordnance  fired  that  can  be  undertaken  in  a  given  period  of  time  with  SIMNET  as  compared 
to  field  exercises.  For  example,  in  an  ll-month  period  in  1987-88,  four  Ml  tank 
simulators  simulated  18,600  miles  of  driving  [65],  which  is  the  average  annual  mileage  for 
23  tanks.  This  means  the  simulators  were  able  to  simulate  approximately  six  times  as  much 
driving  per  time  period  as  the  average  tank  would  be  driven.  In  [611,  tank  platoons 
traveled  approximately  100  miles  in  three  days.  Assuming  each  simulator  was  used  four 
days  a  week  or  200  days  a  year,  this  means  each  simulator  would  be  “driven"  about  7000 
miles,  or  nine  times  the  average  annual  tank  mileage. 

(3)  Effectiveness  of  NTC  Conditioned  on  Home-Station  Training.  One 
way  to  measure  the  marginal  effectiveness  of  home-station  training  is  to  use  the  amount  of 
such  traiiung  to  explain  performance  as  measured  in  other,  more  realistic,  training 
exercises.  One  such  analysis  done  by  the  Army  Research  Institute  (ARI)  is  reported  in 
[66]  and  [67].  In  this  analysis,  the  casualty  exchange  ratio  (percentage  opposing  force 
(OPFOR)  vehicles  killed  divided  by  percentage  blue  force  (BLUFOR  or  friendly)  vehicles 
killed)  in  NTC  exercises  was  explained  by  tank  mileage  driven  by  the  BLUFOR  in  the  six 
months  prior  to  the  exercises,  and  by  a  subjective  rating  of  the  similarity  of  the  terrain  at 
home  station  and  NTC. 

Tlie  result  of  this  analysis  was  that,  for  defensive  missions,  home-station  mileage 
was  positively  related  to  the  casualty  exchange  ratio,  and  that  a  doubling  of  miles  driven 
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increased  the  exchange  ratio  by  roughly  30%  of  its  value  (e.g.,  if  the  ratio  was  0.5  it  would 
become  0.65).20 

Comparable  analyses  of  the  effectiveness  of  SIMNET-based  training  arc  noi 
available.  Once  CCTT  becomes  operational,  an  analysis  that  explains  the  NTC  casur  ity 
exchange  ratio  using  both  home-station  tank  mileage  and  CCTT  time  should  allow 
comparisons  of  the  effectiveness  of  the  two  approaches. 


(4)  Reserve  Component  Effectiveness.  Very  few  comparisons  have  been 
made  between  the  training  reserve  components  currently  receive  versus  the  training  they 
would  receive  with  mobile  CCTT  equipment  and  occasional  access  to  fixed  CCTT 
equipment.  It  is  our  understanding  that  mobile  CCTT  equipment  would  be  much  more 
effective  than  current  practice.  Failure  to  consider  this  additional  use  leads  to  understating 
the  value  of  SIMNET-like  facilities. 


c.  Cost-Effectiveness  of  SIMNET  and  Home-Station  Training 

Some  inferences  can  be  drawn  from  the  cost  and  effectiveness  discussions 
presented  here;  however,  they  do  not  allow  definitive  answers  to  cost-effectiveness 
questions. 

The  cost  data  in  Table  10,  and  the  effectiveness  data  reported  in  Table  12  show  that, 
on  the  margin,  tasks  trainable  on  SIMNET  should  be  trained  there.  However,  as  shown  in 
the  effectiveness  data,  SIMNET  is  not  capable  of  training  all  the  combat-relevant  tasks. 
Tasks  not  trainable  on  SIMNET  must  still  be  trained  at  NTC  or  in  other  field  exercises. 

Data  cited  previously  on  the  marginal  effectiveness  of  simulator-type  iTdining  [15] 
indicate  that  its  incremental  effectiveness  will  decline  as  more  of  this  training  is  done.  If 
this  also  applies  to  SIMNET  and  home  field  exercises,  it  means  that  even  tasks  cost- 
effectively  trained  at  the  margin  by  any  one  training  method  cannot  necessarily  be  trained 
by  that  method  alone.  Generating  information  on  the  incremental  contribution  of  both 
individual  and  networked  simulator  training  would  better  inform  policy  makers  on  the  cost- 
effective  mix  of  training  options. 


This  figure  was  inteipolated  froiii  graphical  material  in  [67],  the  underlying  eqi  ‘ir  was  not  reproduced 
in  the  citation. 


C.  SUMMARY 


In  this  paper,  we  explained  how  we  developed  a  rough  estimate  of  the  cost  of 
OPTEMPO,  one  of  the  primary  ways  collective  training  is  funded  during  peacetime.  In  FY 
1991  dollars,  for  DoD,  this  estimate  is  $21  billion.  Simulations  can  leverage  these 
expenditures  to  increase  their  efficiency  and  improve  readiness  and  effectiveness. 

The  Analyses  subsection  presents  two  types  of  cost-effectiveness  comparisons:  a) 
the  tradeoff  of  flying-hour  OPTEMPO  for  possible  increases  in  simulator  time,  and  b)  a 
comparison  of  the  costs  and  effectiveness  of  SIMNET  with  actual  exercises  at  field  training 
locations.  The  quality  of  the  information  provided  is  not  sufficient  to  provide  detailed 
policy  guidance,  but  does  appear  to  offer  some  preliminary  insights. 

1 .  OPTEMPO  Versus  Flight  Simulators 

Combining  these  representative  costs  found  in  a  review  of  past  studies  and  a 
selection  of  a  few  current  programs  suggests  that  the  cost  side  of  a  simulator/actual 
equipment  comparison  is  that  flight  simulators  are  10%-33%  of  the  costs  of  actual 
equipment  per  hour  trained.  The  relative  cost  of  simulators  is  10%  of  actual  equipment 
when  both  actual  equipment  and  simulators  are  already  in  the  inventory,  the  33%  relative 
cost  applies  if  simulators  must  be  both  procured  and  maintained.  With  respect  to  the 
relative  effectiveness  of  simulators  and  actual  equipment,  TER  comparisons  from  the 
literature  suggest  that  the  majority  (59%)  of  tasks  trained  have  TERs  greater  than  0.33,  and 
94%  have  TERs  greater  than  0.1. 

Combining  this  cost  and  effectiveness  information  means  that  individuals  can  be 
cost-effectively  trained  on  simulator-trainable  tasks  using  simulators  at  current  ratios  of 
simulator  training  to  actual  equipment  training.  Additionally,  there  are  tasks  that,  for 
safety,  environmental,  or  security  reasons,  are  not  currently  trained  usiqg  actual  equipment. 
Many  of  these  tasks  can  be  trained  using  simulators.  | 

An  aggregate  comparison  of  fiybg  OPTEMPO  and  flight  simulator  TERs  offers  the 
possibility  that  about  5%  of  the  flying- hour  budget  could  be  transfiprred  to  simulator 
acquisition  and  operation.  However,  the  savings  from  a  5%  decrease  in  flying  hours  could 
be  used  to  generate  a  15%  to  50%  increase  in  simulator  hours  (based  oti  the  10%  to  33% 
cost  range  discussed  previously).  The  effects  of  those  sorts  of  increased  on  TERs  should 
be  understood  before  still  greater  simulator  utilization  is  considered,)  and  we  believe 
experiments  should  be  undertaken  to  develop  this  understanding.  Existing  information  on 
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TERs  suggests  that  equal  effectiveness  could  be  maintained  without  using  all  of  the  savings 
from  reduced  OPTEMPO  to  fund  additional  simulator-based  training. 

2.  SIMNET,  NTC,  and  Home-Station  Exercises 

We  found  effectiveness  comparisons  between  SIMNET  and  home-station  field 
exercises  and  an  analysis  of  the  incremental  effect  of  home-station  field  exercises  on  NTC 
performance.  Analysis  of  an  effectiveness  comparison  between  SIMNET  and  home-station 
field  training  supports  the  hypothesis  that  SIMNET  is  extremely  effective  in  increasing 
performance  for  SIMNET-trainable  tasks  relative  to  standard  field  training. 

Tradeoff  analyses  were  done  between  vehicle  OPTEMPO  and  networked  simulator 
costs.  They  showed  that  investment  now  in  SIMNET-like  facilities  could  be  repaid  once 
the  facilities  were  fielded  by  a  decrease  of  8%-14%  of  OPTEMPO.  The  14%  figure  is 
similar  to  an  Army  estimate  of  the  vehicle  mileage  that  would  be  substituted  for  by  CCTT. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Simulators  and  training  devices  are  currently  used  for  some  individual  and  crew 
training  of  operators  and  maintainers.  Data  reported  here  indicate  that  more  extensive  and 
intensive  use  of  simulators  and  training  devices  would  be  cost-effective.  These  data  also 
suggest  that  as  more  of  this  sort  of  training  is  done,  costs  and  effectiveness  outcomes 
should  be  monitored  to  determine  how  much  cost-effectiveness  ratios  change  as  a  result  of 
the  increase  in  simulator  and  training  device  use. 

In  the  past,  the  vast  majority  of  collective  training  efforts  have  been  in  field 
exercises  and  training  missions.  They  still  are.  More  recently,  however,  advanced 
technology  has  been  employed  to  enhance  collective  training  in  units  through  capabilities 
such  as  (a)  instrumented  air  combat  maneuvering  ranges,  fleet  operational  training  facilities, 
and  the  National  Training  Center,  (b)  computer-based  models,  simulations,  and  war 
games,  (c)  the  networking  of  simulators  for  training,  as  in  SIMNET,  and  (d)  aircraft 
simulators  and  training  devices  of  various  complexity. 

There  has  been  relatively  little  R&D  devoted  to  coDective  training.  Unit  training  (in 
the  sense  of  the  team  training  of  interacting  crews,  groups,  or  teams  within  or  between 
units)  has  been  relatively  unexplored  in  the  scholarly  literature,  even  by  the  service 
laboratories,  though  this  is  precisely  the  kind  of  readiness  training  that  is  central  to 
preparation  of  military  units  for  combat.  As  indicated  previously,  such  training  does  take 
place  within  the  operational  units  of  the  services,  but  relatively  independent  of  inputs  from 
the  training-technology  R&D  community,  and  in  the  absence  of  consistent,  quantitative, 
and  objective  evaluation. 

With  respect  to  individual  and  crew  simulators,  the  data  and  analyses  presented  in 
Section  IV  suggest  that  relatively  small  changes  in  the  resources  devoted  to  OPTEMPO,  on 
the  order  of  5%,  could  support  relatively  large  changes  in  simulator  usage,  in  the  range  of 
15%  to  50%.  Effectiveness  data  in  the  literature  imply  that  these  resources  could  be  cost- 
effectively  applied  to  simulator  provision  to  enhance  training.  Alternatively,  some  of  these 
resources  could  be  used  for  simulators  and  training  devices  to  maintain  proficiency  while 
reducing  overall  training  costs.  This  evidence  is  strong,  but  not  conclusive,  primarily 
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because  it  is  not  known  how  rapidly  the  incremental  effectiveness  of  simulator  training  will 
decline  if  such  training  is  significantly  increased. 

With  respect  to  networked  simulators,  the  effectiveness  data  suggest  that  SIMNET- 
like  facilities  are  extremely  effective  in  training  a  large  number  of  tasks.  These  SIMNET- 
like  facilities  cat*  also  be  purchased  with  OPTEMPO  reductions  in  the  range  of  8%  to  14%. 
They  appear  to  be  a  good  buy.  Information  that  would  better  support  policymaking  in  this 
area  involves  quantifying  the  interrelationship  of  the  incremental  effectiveness  of  training 
between  SIMNET-Iike  facilit- js,  NTC,  and  home-station  field  exercises. 

B .  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

The  purpose  of  the  analyses  suggested  below  would  be  to  supply  decision  makers 
with  further  information  on  the  cost-effectiveness  of  incremental  changes  in  simulator  use. 

1 .  Individual  and  Crew  Simulators 

Research  is  needed  to  determine  the  proper  balance  between  simulator-based 
training  and  operational  training.  \ 

The  cost-effectiveness  of  simulations  for  individual  operator  and  maintainer  training 
has  been  studied,  and  there  is  a  body  of  literature  in  this  area.  However,  this  issue  needs 

additional  experimental  research  for  several  reasons:  | 

I 

•  The  cost  of  simulators  is  heavily  based  on  digital  electronic  technology,  and  the 
cost  of  this  technology  has  fallen  dramatically  and  will  continue  to  fall  in  the 
future. 

•  Political  and  environmental  changes  of  the  past  few  years  are  likely  to  further 
constrain  the  use  of  actual  equipment  and  exercises  versus  current  operator  and 
maintainer  simulators. 

•  Previous  research  has  indicated  that  the  marginal  impact  of  simulation  on 
performance  can  change  dramatically  depending  on  the  amount  of  simulator 
training. 

•  The  proper  balance  of  simulator  and  operational  training  is  likely  to  vary 
substantially  by  unit  type. 

These  factors  suggest  that  experiments  that  attempt  to  gain  a  better  understanding  of 
the  contribution  of  alternative  mixes  of  simulator  and  operational  training  to  performance 
would  be  worthwhile. 


2.  Networked,  Collective  Simulators 

Quantitative  cost-effectiveness  studies  of  networked  simulators  should  be 
undertaken. 

The  cost-effectiveness  of  networked  simulators  for  collective  training  has  not  been 
thoroughly  studied.  This  lack  of  study  is  partially  because  networked  simulators  are  a 
recent  innovation,  and  partially  because  it  is  much  more  difficult  to  conceptualize,  separate, 
and  measure  the  incremental  effects  of  alternative  training  approaches  in  this  context 

One  approach  to  remedying  this  situation  would  be  to  examine  the  data  on 
performance  in  NTC  and  home-station  field  exercises  that  have  been  collected  by  the  Army 
Research  Institute.  This  could  be  correlated  with  prior  SIMNET  use.  The  question  of 
interest  would  be  what  differential  training  effectiveness  resulted  from  the  SIMNET 
training.  This  would  allow  tasks  to  be  reassigned  to  training  approaches  so  that,  for 
example,  more  time  would  be  available  during  exercises  for  concentration  on  tasks  that  are 
better  trained  in  that  environment 
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COST  ESTIMATE  DETAIL 

This  appendix  presents  details  for  several  cost  estimates  used  in  this  paper.  These 
include  estimates  of  the  Close  Combat  Tactical  Trainer  (CCTT)  and  the  Aviation  Combined 
Arms  Tactical  Trainer  (AVCATT)  and  of  potential  savings  that  could  arise  from  less 
operating  tempo  (OPTEMPO)  and  fewer  field  exercises.  Also,  cost  data  for  National 
Training  Center  (NTC)  exercises  are  provided.  Finally,  a  cost  per  day  of  training  is 
calculated  for  the  NTC  and  a  simulator  network  {SIMNET)-like  facility  in  an  effort  to 
compare  the  two  training  options. 

CCTT  AND  AVCATT 

Current  plans  for  CCTT  and  AVCATT  call  for  research  and  development  (R&D) 
for  the  next  several  years  and  production  in  the  late  1990s.  For  the  purposes  of  estimating 
cost  a  fielded  life  of  15  years  is  assumed.  The  CCTT  will  have  fixed  and  mobile  units;  the 
former  are  mainly  for  use  by  active  forces,  the  latter  mainly  for  use  by  reserve  forces. 

It  is  our  understanding  that  detailed  cost  estimates  have  not  been  done  for 
AVCATT,  and  that  CCTT  cost  estimates  are  being  closely  held  because  of  the  current  stage 
of  the  acquisition  process.  Therefore,  the  expenditure  profiles  presented  here  are  based  on 
rough  estimates  that  both  the  fixed-site  portion  of  CCTT  and  AVCATT  will  cost  less  than 
$1  billion,  that,  in  constant,  undiscounted  dollars,  the  fixed-site  portion  of  CCTT  will  be 
50%  operating  and  support  (O&S)  and  the  rest  divided  between  R&D  and  procurement, 
and  that  the  mobile  CCTT  and  AVCATT  will  take  advantage  of  some  of  the  R&D  and 
procurement  work  done  for  the  fixed-site  CCTT,  so  they  will  have  more  than  50%  O&S 
costs. 

Tables  A-1  through  A-3  display  time-phased  expenditure  profiles.  They  are  shown 
in  constant  1991  dollars,  in  current  dollars  assuming  a  4%  inflation  rate,  and  in  constant, 
discounted  dollars  using  a  10%  discount  rate. 
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Table  A>1.  Fixed-Site  (Active)  CCTT  Costs 


Cost  (Millions) 


Constant  1991 


Current 


Constant  1991, 
Discounted 


Table  A-2.  Mobile  Site  (Reserve)  CCTT  Costs 


Yer 

Cost  (Millions) 

Constant  1991 
Dollars 

Current 

Dollars 

Constant  1991, 
Discounted 
Dollars 

1993 

5 

5.4 

4.1 

1994 

5 

5.6 

3.8 

1995 

5 

5.8 

3.4 

1996 

5 

6.1 

3.1 

1997 

5 

6.3 

2.8 

1998 

75 

98.7 

38.5 

1999 

70 

95.8 

32.7 

2000 

15 

21.3 

6.4 

2001 

15 

22.2 

5.8 

2002 

15 

23.1 

5.3 

2003 

15 

24.0 

4.8 

2004 

15 

25.0 

4.3 

2005 

15 

26.0 

3.9 

2006 

15 

27.0 

3.6 

2007 

15 

28.1 

3.3 

2008 

15 

29.2 

3.0 

2009 

15 

30.4 

2.7 

2010 

15 

31.6 

2.5 

2011 

15 

32.9 

2.2 

2012 

15 

34.2 

2.0 

2013 

15 

35.5 

1.8 

2014 

__2LQ 

_ L2 

Totals 

$400 

$656.5 

$146.1 

Table  A-3.  AVCATT  Costs 


Year 

Cost  (Millions) 

Constant  1991 
Dollars 

Current 

Dollars 

Constant  1991, 
Discounted 
Dollars 

1993 

15 

16.2 

12.4 

1994 

15 

16.9 

11.3 

199.5 

15 

17.5 

10.2 

1996 

15 

18.2 

9.3 

1997 

15 

19.0 

8.5 

1998 

145 

190.8 

14.4 

1999 

140 

191.6 

65.3 

2000 

35 

49.8 

14.8 

2001 

35 

51.8 

13.5 

2002 

35 

53.9 

12.3 

2003 

35 

56.0 

11.2 

2004 

35 

58.3 

10.1 

2005 

35 

60.6 

9.2 

2006 

35 

63.0 

8.4 

2007 

35 

65.6 

■’.6 

2008 

35 

68.2 

6.9 

2009 

35 

70.9 

6.3 

2010 

35 

73.7 

5.7 

2011 

35 

76.7 

5.2 

2012 

35 

79.8 

4.7 

2013 

35 

82.9 

4.3 

2014 

_25 

_ 

_ 12 

Totals 

$900 

$1,483.4 

$329.2 

DRIVING  AND  FLYING-HOUR  COST  TRADEOFFS 

In  the  notional  expenditure  profiles  displayed  in  the  tables,  CCIT  and  AVCATT  are 
in  place  in  the  years  2000-2015.  Therefore,  at  that  time,  driving  and  flying  hours  are 
removed  from  the  OPTEMPO  budgets  to  pay  back  the  investment  in  CCTT  and  AVCATT. 
This  payback  is  set  equal  to  the  constant,  discounted  dollars  of  the  investment  Table  A-4 
presents  the  payback  for  the  fixed-site  CCTT  and  the  AVCATT  in  terms  of  the  vehicle  and 
flight  OPTEMPO  cost 

The  question  to  be  addressed  is:  Where  does  this  money  come  from?  One  answer 
is  to  take  the  funds  from  general  OPTEMPO  as  described  in  Tables  2, 3,  and  5  in  the  main 
text  Table  A-5  shows  the  percentage  of  these  budgets  that  would  be  needed  for  both  the 
fixed-site  (active)  CCTT,  and  for  the  AVCATT. 


\ 
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Table  A-4.  Vehicle  and  Flight  OPTEMPO  Cost  Tradeoffs 

_ Cost  Tradeoff  (Millions) _ 


Vehicle  OPTEMPO _ Flight  OPTEMPO 


Year 

Constant 
1991  Dollars 

Current 

Dollars 

Constant  1991, 
Discounted 
Dollars 

Constant 
1991  Dollars 

Current 

Dollars 

Constant  1991, 
Discounted 
Dollars 

2000 

$100.1 

$142.5 

$42.5 

$92.8 

$132.1 

$39.4 

2001 

100.1 

148.2 

38.6 

92.8 

137.3 

35.8 

2002 

100.1 

154.1 

35.1 

92.8 

142.8 

32.5 

2003 

100.1 

160.3 

31.9 

92.8 

148.6 

29.6 

2004 

100.1 

166.7 

29.0 

92.8 

154.5 

26.9 

2005 

100.1 

173.4 

26.4 

92.8 

160.7 

24.4 

2006 

100.1 

180.3 

24.0 

92.8 

167.1 

22.2 

2007 

100.1 

187.5 

21.8 

92.8 

173.8 

20.2 

2008 

100.1 

195.0 

19.8 

92.8 

180.7 

18.4 

2009 

100.1 

202.8 

18.0 

92.8 

188.0 

16.7 

2010 

100.1 

210.9 

16.4 

92.8 

195.5 

15.2 

2011 

100.1 

219.4 

14.9 

92.8 

203.3 

13.8 

2012 

100.1 

228.1 

13.5 

92.8 

211.4 

12.5 

2013 

100.1 

237.3 

12.3 

92.8 

219.9 

11.4 

2014 

_ mi 

-■,,246.7 

_ 112 

92.8 

■  228.7 

10.4 

Totals 

Sl.501.7 

$2,853.2 

$355.2 

$1,391.8 

$2,644.4 

$329.2 

Table  A'5.  Percentages  of  Vehicle  and 
Flight  OPTEMPO  Needed  for  Payback 


Label 

Vehicle 

Flight 

Budget  (Millions  of  FY  1991  Dollars) 

$1,290 

$1,100 

Miles/Year  or  Hours/Montta 

800  Miles/Year 

14.5  Hours/Montb 

MilesfYear  or  Hours/Montb  Saved 

62.1  Miles 

1.2  Hours 

Annual  Savings  (Millions  of  FY  1991  Dollars) 

$100.1 

$92.8 

Percentage  for  Payback 

7.8% 

8.4% 

Another  approach  is  to  take  these  funds  from  the  OPTEMPO  and  ammunition  funds 
of  the  imits  that  will  make  use  of  the  CCTT.  Table  A-6  shows  these  costs  and  the 
percentage  of  resources  needed,  expressed  as  both  costs  and  miles  driven.  The  costs  per 
hour  for  the  tanks  are  loaded  with  a  proportional  share  of  the  cost  per  hour  of  the  other 
vehicles  in  the  tank  battalion.  Ammunition  is  assumed  to  be  expended  proportional  to 
mileage,  and  the  inclusion  of  ammunition  expenditure  is  another  reason  Table  A*6  differs 
from  A-5.  The  results  from  Tables  A-5  and  A-6  indicate  that  the  OPTEMPO  traded  will  be 
8%  to  14%,  depending  on  the  base  used. 
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Table  A-6.  Armored  and  Mechanized  Infantry  Payback 


Unit  Type 

Tank  Battalion 

Mechanized 

Infantry 

Divisional 

Cavalry 

Regimental 

Cavalry 

Dollars/Mile* 

S166 

$87 

_b 

$202 

Miles/Year 

800 

742 

_b 

1241 

Number  of  Vehicles 

58 

54 

b 

41 

Dollars 

$7.7 

$3.5 

$6.8 

$10.3 

Ammunition*^ 

4.3 

1.2 

2.1 

4.2 

Number  of  Units 

34 

34 

8 

6 

Total  Cost  by  Unit  Type 

$410.1 

$159.1 

$71.1 

$87 

Total  Dollars 

$747.1 

Percentage  Traded 

13.76% 

Dollars/Year 

$100.1 

Miles/Year  on  800-Mile  Base 

110.1 

Notes:  Algebra  not  exact  due  to  rounding.  All  costs  are  in  millions  of  FY  1991  dollars.  This  analysis  was 
aided  by  conversations  with  Doug  Johnson  and  Lee  Paris  of  U.S.  Army  TRADOC  Analysis  Command,  White 
Sands  Missile  Range,  but  they  bear  no  responsibility  for  our  use  of  their  information. 


*  Includes  apportioned  support  units  obtained  primarily  from  the  Training  Resource  Model  (TRM). 

^  Data  not  provided. 

^  Obtained  from  the  Training  ,'v  munition  Management  Information  System  (TAMIS).  SRC  and  unit  title 
used:  17235J430.  Tank  Batui,.  n  (MlAl);  07245L000,  Inf  BN  (Mech);  17395LOOO,  Cavalry  Squadron 
(2ID);  17485L100,  Armored  Cavalry  Squadron,  ACR  1X6. 

COSTS  OF  EXERCISES  AT  NTC 

NTC  costs  will  be  measured  in  terms  of  operating  or  incremental  cost.  Each 
armored  battalion  and  mechanized  infantry  has  the  opportunity  to  use  NTC  on  an 
approximately  18-month  cycle.  The  visit  takes  about  one  month,  and  half  of  that  time  is 
spent  in  the  actual  exercise.  The  rest  of  the  time  is  spent  packing  and  unpacking  equipment 
and  traveling. 

Table  A-7  shows  the  incremental  cost  of  one  exercise  at  NTC.  These  data  are 
shown  in  more  detail  in  Tables  A- 10  to  A- 12.  They  have  been  rounded  to  the  nearest 
quarter  million  dollars.  A  portion  of  the  pay  and  allowances  for  the  Opposition  Force 
(OPFOR)  is  included.  Although  the  OPFOR  personnel  are  receiving  some  training  from 
their  participation,  the  main  reason  they  are  there  is  to  provide  training  for  others.  Results 
are  provided  three  ways.  Subtotal  A  would  be  the  incremental  costs  for  one  rotation  at 
NTC  above  the  cost  of  field  exercises,  assuming  that  the  field  exercises  had  the  same 
amount  of  driving  and  ammunition  usage.  Subtotal  B  or  the  total  cost  can  be  compared  to 
the  cost  of  a  SIMNET-like  facility,  or  to  item  6  alone,  which  is  a  rough  estimate  of  the  cost 
of  a  home-station  field  exercise  of  similar  duration  as  the  NTC  exercise.  The  difference 
between  Subtotal  B  and  the  total  is  OPFOR  pay  and  allowances. 
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Table  A-7.  NTC  Exercise  Costs 


Item 

Caiecorv 

Cost  Millions 

1 

Transport,  Base  to  Railhead  at  NTC 

$3.50 

2 

Transport,  Railhead  to  NTC  and  Back 

1.50 

3 

Tools  and  Spares 

0.50 

4 

Range  Decontamination 

0.25 

5 

OPFOR  OPTEMPO  and  Ammunition 

4.50 

Subtotal  A 

10.25 

6 

BLUFOR  OPTEMPO  and  Ammunition 

3.00 

Subtotal  B 

13.25 

7 

OPFOR  Pay  and  Allowances 

3.00 

Total 

$16.25 

Sources:  Dr.  Howard  McFann,  ARl,  for  items  1,3.  and  4;  Capt.  Dinan  (DPTVtSEC) 
for  item  2.  For  item  S,  see  Table  A-11.  p.  A-9;  item  6,  Table  A*12,  p.  A-10;  item  7, 
see  p.  A'lO  of  this  appendix. 


COST  PER  DAY  OF  TRAINING 

The  NTC  facility  offers  an  average  of  10  days  of  training  per  battalion  per  year. 
The  SIMNET  facility  is  being  sized  for  20  days  of  training  .'or  each  person  in  the  battalion 
for  each  year.  The  availability  of  SIMNET  equipment  would  mean  that,  for  those  tasks 
trainable  through  SIMNET,  units  could  practice  until  they  goi  ii  right,  and  slow  learners 
would  not  slow  down  fast  learners.  Selective,  intensive  training  such  as  this  would 
increase  overall  unit  readiness  in  a  way  not  possible  without  additional  NTCs  and  more 
time  spent  there,  and  the  low  variable  cost  of  simulator  networks  would  not  be  available 
even  with  additional  NTCs. 

Another  way  to  compare  these  options  is  to  use  the  cost  data  to  construct  a  co.^t  per 
training  day  per  year  measure  of  the  cost  of  training  through  these  various  options.  This 
measure  simply  shows  what  the  present  value  of  one  day  of  training  would  cost  for  the 
entire  force.  Table  A-8  displays  this  measure  and  indicates  whether  the  cost  is  variable  or 
total. 


Tabl«  A*8.  Cost  p«r  Training  Day  per  Year 


Option 

Cost  (Millions  of 
Discounted,  FY 

1991  Dollars) 

Variable 
or  Total 

NTC  with  Pay  and  Allowances 

$4.6 

Variable 

NTC  without  Pay  and  Allowances 

3.8 

Variable 

CCTT  and  AVCTT,  20  days 

2.8 

Total 

CCTT,  20  days 

1.7 

Total 

\  •  .  '  < 


As  previously  mentioned,  specific  rese'  e  v'^mponent  trainers  are  also  being 
acquired,  and  reserve  units  may  be  able  to  use  the  equipment  during  the  times  that  active 
units  are  not  using  it.  The  reserve  component  trainers  would  be  an  extra  cost  and  generate 
additional  training,  reserve  use  of  the  active  force  CCI  installations  would  generate  little 
extra  cost,  but  extra  benefits.  These  cost  estimates  are  not  included  here. 

Another  approach  to  understanding  these  data  involve  generating  relative  cost 
ratios.  For  example,  Table  A-9  shows  the  ratio  Psimnet/Pntc  for  the  various  cost  and 
days  of  training  assumptions  discussed  previously.  Using  the  “CCTT  and  AVCTT’ 
section  of  the  table,  the  SIMNET-like  facility  costs  between  60%  and  73%  of  NTC  per  day 
of  training,  using  CCTT  alone  shows  36%  to  44%  of  the  cost  per  day  at  NTC. 


Table  A-9.  Cost  Ratios  for  SIMNET 
and  NTC  Training  Options 


NTC 

CCTT  and 

Averr 

(20  Davsl 

CCTT 
(20  Days) 

Without  Pay  and  Allowances 

0.73 

0.44 

With  Pay  and  Allowances 

0.60 

0.36 

Note:  The  costs  used  ire  the  viriable  cost  of  NTC  and  the  total  cost  of  a 
SIMNET-like  facility. 


These  cost  figures  are  biased  against  SIMNET-like  facilities  in  several  ways.  As 
mentioned  previously,  u^ning  of  reserve  component  at  these  sites  is  ignored.  The  CCTT 
costs  include  installations  in  Europe  and  Korea,  but  the  Combined  Arms  Training  Centers 
in  those  areas  are  not  counted  in  the  costs  of  NTC.  In  addition,  the  opportunity  cost  of  the 
land  on  which  NTC  and  home-station  training  take  place  is  not  included.  This  is  probably 
not  significant  for  NTC  itself,  but  is  clearly  becoming  more  important  for  other  training  and 
exercise  areas  (see  Reference  [8]),  and  would  be  a  large  part  of  the  cost  of  another  NTC. 

NTC  COSTS  FOR  COMPARISON  WITH  SIMNET-LIKE  FACILITY 

Table  A-10  shows  the  Subtotal  B  and  total  from  Table  A-7  annualized  and 
displayed  in  current  and  constant  1991,  discounted  dollars  for  the  years  2(XX)-2015. 

INCREMENTAL  OPTEMPO  COSTS  FOR  NTC  VEHICLES 

Most  of  the  incremental  costs  for  NTC,  along  with  their  sources,  are  listed  in 
Table  A-7.  The  details  of  the  OPTEMPO  cost  estimate  for  OPFOR  are  shown  in  Table  A- 
1 1.  The  M551  (Sheridan)  masquerades  as  an  OPFOR  vehicle. 
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Table  A>10.  NTC  Costs  with  and  without  OPFOR  Pay 


Cost  (Millions) 


NTC  Costs  without  OPFOR  Pay 

Constant 


Year 

Constant 

1991 

Current 

Dollars 

1991, 

Discounted 

2000 

$159.0 

$226.3 

$67.4 

2001 

159.0 

235.4 

61.3 

2002 

159.0 

244.8 

55.7 

2003 

159.0 

254.6 

50.7 

20W 

159.0 

264.7 

46.1 

2005 

159.0 

275.3 

41.9 

2006 

159.0 

286.4 

38.1 

2007 

159.0 

297.8 

34.6 

2008 

159.0 

309.7 

31.5 

2009 

159.0 

322.1 

28.6 

2010 

159.0 

335.0 

26.0 

2011 

159.0 

348.4 

23.6 

2012 

159.0 

362.3 

21.5 

2013 

159.0 

376.8 

19.5 

2014 

_ I52il 

_ 22La 

_ iia 

Totals 

$2,385.0 

$4,531.5 

$564.2 

NTC  Costs  with  OPFOR  Pay 


Constant 

1991 

Current 

Dollars 

Constant 

1991, 

Di.scounted 

$195.0 

$277.5 

$82.7 

195.0 

288.6 

75.2 

195.0 

300.2 

68.3 

195.0 

312.2 

62.1 

195.0 

324.7 

56.5 

195.0 

337.7 

51.3 

195.0 

351.2 

46.7 

195.0 

365.2 

42.4 

195.0 

379.8 

38.6 

195.0 

395.0 

35.1 

195.0 

410.8 

31.9 

195.0 

427.3 

29.0 

195.0 

444.4 

26.4 

195.0 

462.1 

24.0 

_ 195.0 

_ mfi 

_ 212 

$2,925.0 

$5,557.5 

$691.9 

Table  A*11.  OPFOR  OPTEMPO  and  Ammunition  Cost 


Unit  Type 

OPFOR 

Dollars  per  Mile* 

$82 

Miles  per  Ycat* 

3^00 

Number  of  Vehicles® 

193 

Annual  Expenditure 

$50.6 

Monthly  OPTEMPO  Expenditure 

$4.22 

Monthly  Ammunition  Expenditure 

$0.27 

Total  Monthly  Expenditure 

$4.49 

Note:  Except  for  dollars  per  mile,  all  costs  are  in  millions 
of  FY  1991  doUars. 

•  Anny  Average  OPTEMPO  for  M551. 

**  Reference  (67]  shows  approximately  4,000  miles  per 
year,  but  800  would  occur  reganiless. 

*  According  to  Major  Centiick,  DSOPS  Training,  the 
number  fluctuates  between  193  and  198. 
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INCREMENTAL  BLUFOR  COSTS  FOR  NTC  EXERCISES 

Table  A- 12  shows  the  monthly  OPTEMPO  costs  for  the  blue  force  (BLUFOR  or 
friendly  forces)  OPTEMPO  costs.  With  the  exception  of  mileage  driven,  it  recasts 
information  from  Table  A- 6  in  the  form  used  for  Table  A-1 1.  Mileage  driven  is  based 
panially  on  a  monthly  average  annual  OPTEMPO  mileage,  e.g.,  800  miles  for  tank 
battalions.  In  addition,  because  time  at  NTC  is  the  culmination  of  a  great  deal  of  effort,  we 
believe  that  commanders  will  save  OPTEMPO  budget  in  order  to  expend  it  at  NTC. 
Therefore,  we  assume  that  mileage  will  accrue  at  a  rate  50%  higher  than  average,  e.g., 
1 ,200  miles  per  year  for  tank  battalions. 


Table  A*12.  BLUFOR  OPTEMPO  and  Ammunition  Costs 


Unit  Type 

Tank  Battalion 

Mechanized 

Infantry 

Dollars  per  Mile 

$166 

$87 

Miles  per  Year 

uoo 

1,113 

Number  of  Vehicles 

58 

54 

Annual  Expenditure 

$11.6 

$5.3 

Monthly  Expenditure 

$0,965 

$0,438 

Total  Monthly  OPTEMPO  Expenditure 

$1,403 

Total  Monthly  Ammunition  Experiditure 

1.684 

Total  Monthly  Expenditure 

$3,087 

Notes:  Except  for  dollars  per  mile,  all  costs  are  in  millions  of  FY  1991  dollars. 
Cost  per  hour  and  number  of  vehicles  from  Table  A-6,  miles  per  year  from  estimate 
in  text,  ammunition  usage  from  Training  Ammunition  Manager  (AFZJ-PTT)  NTC, 
Fort  Irwin. 


OPFOR  MANPOWER  COSTS 

The  assumptions  made  and  calculations  used  for  OPFOR  manpower  costs  are 
explained  here.  The  combat  strength  of  the  OPFOR  is  2,050,  consisting  of  1,600 
permanently  assigned  personnel  and  450  augmentees  (330  infantry  and  120  engineers). 
We  assumed  that  augmentee  participation  is  training;  hence,  they  are  not  counted  in  the  cost 
estimate.  We  further  assumed  that  the  permanently  assigned  OPFOR  is  receiving  training 
as  well,  but  far  more  training  than  it  otherwise  would.  Therefore,  the  cost  estimate  is 
decremented  by  25%.  A  value  of  $30  thousand  per  year  per  person  is  used  as  the  average 
annual  pay  and  allowances.  This  results  in  $3  million  per  month  in  OPFOR  personnel 
costs  allocated  against  NTC  ($30,00th  x  1,600  personnel  =  $48,000,000  per  year, 
$48,000,000  X  0.75  =  $36,000,000  per  year,  $36,000,000  +  12  months  =  $3,000,000  p’r 
month). 
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APPENDIX  B 

STATISTICAL  DATA  AND  ANALYSIS 


This  appendix  summarizes  the  data  that  compare  the  effects  of  SIMNET  and  field 
exercises  training  on  the  performance  of  specific  field  exercises.  It  then  describes  the 
statistical  test  used  and  presents  the  results  of  those  tests. 

DATA 

The  data  came  from  Gound  and  Schwab  [62].  The  U.S.  Army  Armor  and 
Engineering  Board  generated  a  control  group  and  an  experimental  group  of  tank  platoons, 
gave  them  a  pre-test  to  determine  their  competence,  trained  the  experimental  group  on 
SIMNET  and  the  control  group  through  standard  field  training,  and  then  tested  the  groups 
after  the  training.  The  exercises  used  were  taken  from  the  Army  Training  and  Evaluation 
Program  17-237-10  Mission  Training  Plan  (coordinating  draft)  situation  training  exercises 
(STX).  The  exercises  chosen  were  B  (Hasty/Deliberate  Defense),  E  (Movement  to 
Combat),  and  F  (Hasty  Attack).  Individual  tasks  within  each  exercise  were  rated  on 
whether  they  were  fully,  partially,  or  not  trainable  on  SIMNET. 

The  groups  were  scored  “Go”  or  “No  Go”  on  the  tasks  within  the  given  exercises. 
One  summarization  of  these  results  is  presented  in  Table  11  in  the  main  text.  That 
summarization  reveals  a  problem  with  the  design  of  the  experiment,  but  hides  one  way  to 
analyze  the  data  that  reveals  additional  information. 

The  problem  with  the  experiment  is  that  the  SIMNET-trained  units  were  more 
capable  at  the  start  of  the  test.  Therefore,  it  is  possible  that  the  differential  results  from 
SIMNET  and  home-station  exercises  arc  a  product  of  characteristics  of  the  units  rather  than 
of  the  training.  In  addition,  these  percentages  include  tasks  that  were  scored  only  pre- 
training  or  only  post-training. 

These  data  were  reanalyzed  at  IDA.'  In  this  analysis,  we  took  advantage  of  the 
fact  that  the  data  identified  three  kinds  of  tasks  being  tested;  those  that  were  judged  fully, 
partially,  and  not  trainable  by  SIMNET.  We  also  removed  all  data  that  did  not  have  both  a 


'  Additional  statistical  expertise  and  effort  were  contributed  by  Philip  M.  Lurie  of  IDA. 
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pre-  and  post-test  score.  These  data  arc  summarized  in  the  contingency  tables  shown  in 
Table  B-1.  Each  of  the  six  sub-tables  represents  data  from  either  the  experimental 
(SIMNET)  or  control  (field  exercise)  group,  on  tasks  that  are  either  fully,  partially,  or  not 
trainable  by  SIMNET. 


Table  B-1.  Contingency  Tables  for  Field  Exercise  and  SIMNET  Training 


Partially  Trainable 


Not  Trainable 


Go 

No  Go  Total 

% 

Go 

No  Go 

Total 

Percent 

Go 

No  Go 

Total 

% 

SIMNET 

24 

92 

Go 

47 

6  53 

58 

26 

4 

3b 

100 

20 

4 

No  Go 

25 

13  38 

42 

0 

0 

0 

0 

1 

1 

2 

8 

Total 

72 

19  91 

26 

4 

30 

21 

5 

26 

Percent 

78 

22 

87 

13 

81 

19 

Field  Excerise 

25 

81 

Go 

26 

18  44 

44 

22 

2 

24 

86 

21 

4 

No  Go 

21 

35  56 

56 

2 

2 

4 

14 

4 

2 

6 

19 

Total 

47 

53  100 

24 

4 

28 

25 

6 

31 

Percent 

47 

53 

86 

14 

81 

19 

For  each  of  the  six  combinations,  a  sub-table  shows  a  cross-tabulation  of  the  pre- 
and  post -test  score  on  the  tasks  within  the  individual  exercises.  The  table  should  be  read  as 
follows.  Using  the  data  for  the  experimental  (SIMNET)  units  on  tasks  that  were  judged  to 
be  fully  trainable  on  SIMNET,  there  were  91  observations  on  a  particular  task  for  a 
particular  platoon  that  had  a  pre-  and  post-test  grade.  Of  those  observations,  47  were  Go 
both  before  and  after  the  test,  25  were  No  Go  before  the  test  and  Go  after  the  SIMNET 
training,  13  were  No  Go  both  before  aiid  after  the  training,  and  6  were  Go  before  the 
training,  and  No  Go  afterward.  One  way  of  viewing  these  data  is  that  58%  of  the  pre¬ 
training  scores  were  Go,  and  78%  of  the  post-training  scores  were  Go.  Another  way  is 
that  6  of  the  53  pre-training  Gos  became  No  Gos,  while  25  of  the  38  pre-training  No  Gos 
became  Gos. 

The  questions  are  whether  both  types  of  training  at  least  maintain  competence,  and 
whether  the  SIMNET  training  is  better  than  field  training  at  maintaining  or  improving 
competence. 

STATISTICAL  TEST 

Two  generalizations  of  each  of  the  sub-tables  in  Table  B-1  are  shown  in  Table  B-2. 
In  Table  B-2,  a,  b,  c,  d,  and  n  are  the  observed  cell  frequencies,  and  pa,  pb,  pc,  and  pd  are 
the  probabilities  derived  by  diviefing  the  observed  ficquencies  by  the  total  sample,  n. 
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Table  B-2.  Generalized  Contingency  Tables 


Frequencies 

Probabilities 

Go  No  Go  Total 

Go  No  Go  Total 

Go 

a 

b 

a-f  b 

pa 

pb 

pa  +  pb 

No  Go 

c 

d 

c  +  d 

pc 

Pd 

pc  +  pd 

Total 

a  +  c 

b  +  d 

n 

pa  +  pc 

pb  +  pd 

1 

The  null  hypothesis  to  be  tested  is  the  probability  that  pb  =  pc.  The  alternative 
hypothesis  is  that  pb  ^  pc.  If  the  null  hypothesis  is  accepted,  the  training  has  had  no 
significant  effect  on  performance.  If  the  alternative  hypothesis  is  accepted,  the  training  has 
had  a  significant  effect  on  performance.  If  the  alternative  hypothesis  is  accepted,  and  pc  > 
pb,  then  the  training  has  been  helpful. 

The  test  to  be  used  is  called  McNemar’s  test  (for  descriptions  of  this  test,  see 
References  [68],  p.  72-77,  or  [69],  p.  257-261).  The  test  statistic  is: 

2  (|b-c|-l)^ 

^  ■  (b  +  c) 

distributed  chi  square  with  one  degree  of  freedom.  The  critical  values  for  confidence  levels 
0.90,  0.95,  and  0.99  are  2.71,  3.84.  and  6.63,  respectively.  The  calculated  results  are 
shown  in  Table  B-3.  As  can  be  seen  from  the  table,  SIMNET  training  on  tasks  that  were 
judged  fully  trainable  by  SIMNET  significandy  improved  post-training  performance.  None 
of  the  other  effects  were  significandy  different  from  zero.  While  the  initial  difference  in  the 
experimental  and  control  group  suggest  caution,  this  finding  is  consistent  with  SIMNET 
being  a  relatively  effective  training  method. 

Table^  B-3.  Calculated  Test  Statistic  Values 

Fully  Trainable  Partially  Trainable  Not  Trainable 
SIMNET  1  10.45  2.25  0.80 

Field  Trained  I  0.10 _ 025 _  0.13 
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IB 


ABBREVIATIONS 


ADST 

Advanced  Distributed  Simulation  Technology 

ASD(FM&P) 

Assistant  Secretary  of  Defense  (Foice  Management  and  Personnel) 

AVCATT 

Aviation  Combined  Arms  Tactical  Trainer 

CATS 

Combined  Arms  Training  Strategy 

CCTT 

Close  Combat  Tactical  Trainer 

CGTU 

crew,  group,  team,  or  unit 

CINC 

commander-in-chief 

CONUS 

continental  United  States 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DoD 

Deparunent  of  Defense 

DSB 

Defense  Science  Board 

GAO 

Government  Accounting  Office 

IDA 

Institute  for  Defense  Analyses 

MOE 

measure  of  effectiveness 

NTC 

National  Training  Center 

O&S 

operating  and  support 

OASD(FM&P) 

Office  of  the  Assistant  Secreta:  ■  of  Defense  (Force  Management  and 
Personnel) 

OJT 

on-the-job  training 

OPFOR 

Opposition  Force 

OPTEMPO 

operating  tempo 

OSD 

Office  of  the  Secretary  of  Defense 

PM  TRADE 

Project  Manager,  Training  Devices 

R&D 

research  and  development 

RDT&E 

research,  development,  test  and  evaluation 

SECDEF 

Secretary  of  Defense 

SIMNET 

simulator  networking 

SOF-ATS 

Special  Operations  Forces-Air  Training  System 

STX 

situation  training  exercises 

TER 

Transfer  Effectiveness  Ratio 

TRADOC 

Training  Doctrine  and  Command 
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